Sep 10 15:27:09.101990 ip-10-0-141-82 systemd[1]: Starting Kubernetes Kubelet... Sep 10 15:27:09.490135 ip-10-0-141-82 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:27:09.490135 ip-10-0-141-82 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 15:27:09.490135 ip-10-0-141-82 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:27:09.490135 ip-10-0-141-82 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 15:27:09.490135 ip-10-0-141-82 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:27:09.491067 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.490991 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 15:27:09.494178 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494163 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:27:09.494178 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494177 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494181 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494184 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494186 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494193 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494197 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494199 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494202 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494205 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494208 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494211 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494213 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494216 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494219 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494221 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494224 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494227 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494229 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494232 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494234 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:27:09.494242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494237 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494240 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494242 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494245 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494248 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494250 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494253 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494256 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494258 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494261 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494264 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494266 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494269 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494271 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494274 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494276 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494279 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494282 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494286 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494289 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:27:09.494850 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494293 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494296 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494299 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494301 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494304 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494306 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494308 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494311 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494313 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494316 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494318 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494321 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494323 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494326 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494329 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494332 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494335 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494337 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494340 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494343 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:27:09.495566 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494345 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494349 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494353 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494355 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494358 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494361 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494363 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494365 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494368 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494371 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494374 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494377 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494381 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494384 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494387 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494389 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494392 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494394 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494397 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:27:09.496088 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494399 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494402 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494404 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494407 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494409 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494413 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494415 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494419 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494422 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494424 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494427 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494430 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494432 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494435 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494437 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494440 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494443 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494445 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:27:09.496539 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.494447 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:27:09.497071 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497055 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:27:09.497071 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497069 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497074 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497077 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497080 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497084 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497087 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497089 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497092 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497095 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497097 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497100 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497102 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497106 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497108 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497110 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497113 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497116 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497118 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497121 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:27:09.497128 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497124 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497126 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497129 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497132 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497135 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497137 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497141 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497144 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497147 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497149 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497152 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497155 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497161 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497164 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497167 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497169 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497171 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497174 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497177 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497179 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:27:09.497644 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497182 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497184 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497187 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497190 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497193 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497195 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497198 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497201 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497203 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497205 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497208 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497211 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497213 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497216 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497221 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497225 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497228 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497230 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497233 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497236 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:27:09.498137 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497238 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497242 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497245 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497248 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497250 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497253 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497255 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497258 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497260 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497263 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497265 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497267 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497270 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497272 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497275 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497277 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497280 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497283 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497287 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497289 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:27:09.498623 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497292 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497295 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497298 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497301 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497304 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497308 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497312 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497316 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497319 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497322 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497325 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497328 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497330 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497333 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497336 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497339 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497342 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497345 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497347 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497711 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:27:09.499120 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497717 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497720 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497723 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497725 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497729 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497731 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497734 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497737 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497739 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497742 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497745 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497749 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497753 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497756 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497758 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497762 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497765 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497768 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497770 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:27:09.499603 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497773 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497775 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497778 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497781 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497785 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497788 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497792 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497794 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497797 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497800 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497802 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497805 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497807 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497810 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497813 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497815 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497818 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497820 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497823 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497825 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:27:09.500093 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497828 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497831 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497833 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497836 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497839 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497841 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497844 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497846 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497849 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497853 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497855 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497858 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497861 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497864 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497867 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497869 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497872 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497874 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497877 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:27:09.500590 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497879 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497882 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497884 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497887 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497889 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497892 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497894 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497897 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497900 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497902 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497905 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497907 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497910 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497912 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497915 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497918 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497923 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497926 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497929 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497932 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:27:09.501066 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497935 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497938 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497940 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497943 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497946 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497949 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497951 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497954 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497956 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497959 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497961 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497964 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497967 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497969 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497972 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497974 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497977 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497980 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497982 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.497984 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:27:09.501544 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498049 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498083 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498090 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498094 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498098 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498101 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498106 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498111 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498114 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498117 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498121 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498124 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498127 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498130 2581 flags.go:64] FLAG: --cgroup-root="" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498133 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498136 2581 flags.go:64] FLAG: --client-ca-file="" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498140 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498143 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498147 2581 flags.go:64] FLAG: --cluster-domain="" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498150 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498154 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498157 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498160 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498164 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 15:27:09.502019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498167 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498170 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498173 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498176 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498180 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498183 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498186 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498189 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498193 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498196 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498198 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498201 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498204 2581 flags.go:64] FLAG: --enable-server="true" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498207 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498212 2581 flags.go:64] FLAG: --event-burst="100" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498215 2581 flags.go:64] FLAG: --event-qps="50" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498218 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498221 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498225 2581 flags.go:64] FLAG: --eviction-hard="" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498229 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498232 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498236 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498239 2581 flags.go:64] FLAG: --eviction-soft="" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498242 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498245 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 15:27:09.502583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498251 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498254 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498257 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498260 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498263 2581 flags.go:64] FLAG: --feature-gates="" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498269 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498272 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498275 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498278 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498281 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498284 2581 flags.go:64] FLAG: --help="false" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498287 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498290 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498293 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498296 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498299 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498303 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498310 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498313 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498316 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498319 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498322 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498325 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498328 2581 flags.go:64] FLAG: --kube-reserved="" Sep 10 15:27:09.503232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498331 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498333 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498336 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498339 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498342 2581 flags.go:64] FLAG: --lock-file="" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498345 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498348 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498351 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498357 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498360 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498363 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498366 2581 flags.go:64] FLAG: --logging-format="text" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498369 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498372 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498375 2581 flags.go:64] FLAG: --manifest-url="" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498378 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498382 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498384 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498388 2581 flags.go:64] FLAG: --max-pods="110" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498391 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498394 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498397 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498400 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498403 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498406 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 15:27:09.503909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498410 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498417 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498420 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498423 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498426 2581 flags.go:64] FLAG: --pod-cidr="" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498429 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498434 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498438 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498441 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498444 2581 flags.go:64] FLAG: --port="10250" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498447 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498450 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05bc6b36dbc657853" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498454 2581 flags.go:64] FLAG: --qos-reserved="" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498457 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498459 2581 flags.go:64] FLAG: --register-node="true" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498462 2581 flags.go:64] FLAG: --register-with-taints="" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498466 2581 flags.go:64] FLAG: --registry-burst="10" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498469 2581 flags.go:64] FLAG: --registry-qps="5" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498472 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498475 2581 flags.go:64] FLAG: --reserved-memory="" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498478 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498481 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498485 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498488 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498491 2581 flags.go:64] FLAG: --runonce="false" Sep 10 15:27:09.504521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498493 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498497 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498500 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498503 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498506 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498509 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498512 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498516 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498520 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498523 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498526 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498529 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498532 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498535 2581 flags.go:64] FLAG: --system-cgroups="" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498538 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498543 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498546 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498549 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498552 2581 flags.go:64] FLAG: --tls-min-version="" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498555 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498558 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498561 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498564 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498567 2581 flags.go:64] FLAG: --v="2" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498571 2581 flags.go:64] FLAG: --version="false" Sep 10 15:27:09.505140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498575 2581 flags.go:64] FLAG: --vmodule="" Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498579 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.498582 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498703 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498708 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498711 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498714 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498718 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498721 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498724 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498727 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498731 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498734 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498737 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498741 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498745 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498748 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498751 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498753 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:27:09.505767 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498756 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498759 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498762 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498765 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498767 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498770 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498774 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498777 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498779 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498782 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498785 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498788 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498792 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498794 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498798 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498801 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498804 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498806 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498809 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498811 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:27:09.506235 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498814 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498816 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498819 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498821 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498824 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498827 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498829 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498831 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498835 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498838 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498840 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498843 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498846 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498848 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498850 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498853 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498855 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498858 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498862 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498864 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:27:09.506775 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498867 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498869 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498873 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498876 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498879 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498882 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498885 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498888 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498891 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498894 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498897 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498900 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498902 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498905 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498908 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498910 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498913 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498915 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498918 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:27:09.507266 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498920 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498924 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498926 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498929 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498931 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498934 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498937 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498939 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498942 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498944 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498947 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498951 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498953 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498956 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498958 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498961 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498963 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498966 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498968 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498971 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:27:09.507751 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498973 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498976 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498979 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.498982 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.499716 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.506427 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.506442 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.506461 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.506467 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506539 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506544 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506548 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506552 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506554 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506558 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:27:09.508241 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506560 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506563 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506565 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506568 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506571 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506573 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506576 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506579 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506582 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506584 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506588 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506590 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506594 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506598 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506601 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506605 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506608 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506612 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506615 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:27:09.508613 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506618 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506620 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506623 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506626 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506630 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506633 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506636 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506639 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506642 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506645 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506648 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506650 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506653 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506655 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506658 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506676 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506680 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506684 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506687 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506690 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:27:09.509081 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506692 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506695 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506698 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506700 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506703 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506705 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506708 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506711 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506713 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506716 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506718 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506721 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506723 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506726 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506728 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506731 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506733 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506737 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506739 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:27:09.509568 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506742 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506744 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506747 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506749 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506752 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506754 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506757 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506759 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506762 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506764 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506767 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506769 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506771 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506774 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506777 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506780 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506782 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506785 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506788 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:27:09.510171 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506792 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506796 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506798 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506801 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506803 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506806 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506808 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506810 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506813 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506815 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506818 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506820 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506824 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506826 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506829 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:27:09.510615 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506831 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.506836 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506953 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506958 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506961 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506964 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506967 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506970 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506972 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506975 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506977 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506979 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506982 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506984 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:27:09.510991 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506987 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506989 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506992 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506994 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506997 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.506999 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507002 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507004 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507007 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507009 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507012 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507014 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507016 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507019 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507021 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507024 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507027 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507029 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507032 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507034 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:27:09.511339 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507037 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507040 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507042 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507045 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507047 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507050 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507052 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507054 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507057 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507060 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507063 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507067 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507070 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507072 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507075 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507078 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507081 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507083 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507086 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:27:09.511831 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507089 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507093 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507096 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507098 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507100 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507103 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507105 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507108 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507111 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507113 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507116 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507119 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507121 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507124 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507127 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507129 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507132 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507134 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507137 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507139 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:27:09.512294 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507142 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507144 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507147 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507149 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507151 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507154 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507157 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507159 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507162 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507164 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507167 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507169 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507172 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507174 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507177 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507179 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507183 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507187 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507189 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507192 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:27:09.512798 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507194 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507197 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507199 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507202 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507205 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507207 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507210 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:09.507212 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.507217 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.507316 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.509238 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.510214 2581 server.go:1019] "Starting client certificate rotation" Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.510315 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:27:09.513386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.511138 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:27:09.513815 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.512878 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:27:09.518374 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.518361 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:27:09.530909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.530889 2581 log.go:25] "Validated CRI v1 runtime API" Sep 10 15:27:09.533588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.533573 2581 log.go:25] "Validated CRI v1 image API" Sep 10 15:27:09.534451 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.534433 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 15:27:09.540860 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.540840 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:27:09.542163 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.542145 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 98d2d9bc-1e0e-4162-971a-836a820f20f7:/dev/nvme0n1p3 d53ab3d9-71e8-42d5-b0a9-b30674a7cfca:/dev/nvme0n1p4] Sep 10 15:27:09.542224 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.542162 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 15:27:09.547879 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.547767 2581 manager.go:216] Machine: {Timestamp:2026-09-10 15:27:09.546391214 +0000 UTC m=+0.329936028 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100961 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca4f365c25e07b0b630717b104592 SystemUUID:ec2ca4f3-65c2-5e07-b0b6-30717b104592 BootID:3e9aed16-2c0b-444f-a524-f8f874b1b448 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:71:c0:18:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:71:c0:18:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:00:c8:dd:3f:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 15:27:09.547879 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.547872 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 15:27:09.547990 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.547958 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 15:27:09.551451 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.551424 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 15:27:09.551601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.551453 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 15:27:09.552633 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.552623 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 15:27:09.552677 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.552636 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 15:27:09.552677 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.552644 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:27:09.553261 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.553251 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:27:09.554422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.554412 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 15:27:09.557174 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557162 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:27:09.557283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557273 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 15:27:09.557321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557304 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 10 15:27:09.557321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557319 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 15:27:09.557391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557339 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 15:27:09.557391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557349 2581 kubelet.go:398] "Adding apiserver pod source" Sep 10 15:27:09.557391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557357 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 15:27:09.557720 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557709 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:27:09.557760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.557728 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:27:09.560052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.560030 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 15:27:09.562231 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562214 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 15:27:09.562296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562255 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 15:27:09.562402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562392 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 15:27:09.562435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562406 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 15:27:09.562435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562413 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 15:27:09.562435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562418 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 15:27:09.562435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562425 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 15:27:09.562435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.562431 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 15:27:09.563076 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563067 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 15:27:09.563106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563077 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 15:27:09.563106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563084 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 15:27:09.563106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563090 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 15:27:09.563106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563098 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 15:27:09.563106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563106 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 15:27:09.563985 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.563969 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mkbl" Sep 10 15:27:09.564777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.564762 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 15:27:09.564777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.564778 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 15:27:09.565982 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.565957 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 15:27:09.566292 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.565994 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 15:27:09.569318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.569304 2581 server.go:1268] "Started kubelet" Sep 10 15:27:09.569418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.569392 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 15:27:09.569474 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.569399 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 15:27:09.569538 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.569487 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 15:27:09.570125 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.570097 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 15:27:09.570194 ip-10-0-141-82 systemd[1]: Started Kubernetes Kubelet. Sep 10 15:27:09.571174 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.571159 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 10 15:27:09.571364 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.571350 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 15:27:09.572831 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.572810 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mkbl" Sep 10 15:27:09.574857 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.574840 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 15:27:09.576369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.576349 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 15:27:09.576902 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.576884 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 15:27:09.576993 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.576978 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:09.577645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577629 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 15:27:09.577645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577629 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 15:27:09.577791 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577654 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 15:27:09.577791 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577716 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 10 15:27:09.577791 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577726 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 10 15:27:09.577883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577836 2581 factory.go:153] Registering CRI-O factory Sep 10 15:27:09.577883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577863 2581 factory.go:223] Registration of the crio container factory successfully Sep 10 15:27:09.577935 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577905 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 15:27:09.577935 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577911 2581 factory.go:55] Registering systemd factory Sep 10 15:27:09.577935 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577917 2581 factory.go:223] Registration of the systemd container factory successfully Sep 10 15:27:09.577935 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577929 2581 factory.go:103] Registering Raw factory Sep 10 15:27:09.578108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.577938 2581 manager.go:1196] Started watching for new ooms in manager Sep 10 15:27:09.578316 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.578304 2581 manager.go:349] Starting recovery of all containers Sep 10 15:27:09.579684 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.579651 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:09.584145 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.583880 2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-82.ec2.internal\" not found" node="ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.589906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.589891 2581 manager.go:354] Recovery completed Sep 10 15:27:09.594439 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.594383 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.596592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.596574 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.596694 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.596609 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.596694 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.596623 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.597306 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.597288 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 15:27:09.597306 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.597304 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 15:27:09.597432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.597319 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.597432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.597324 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:27:09.598058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.598041 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.598110 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.598077 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.598110 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.598094 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.677500 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.677458 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:09.742059 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.741578 2581 policy_none.go:49] "None policy: Start" Sep 10 15:27:09.742059 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.741600 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 15:27:09.742059 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.741612 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 15:27:09.742709 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.742691 2581 policy_none.go:47] "Start" Sep 10 15:27:09.772204 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772186 2581 manager.go:337] "Starting Device Plugin manager" Sep 10 15:27:09.772280 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.772227 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 15:27:09.772280 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772241 2581 server.go:87] "Starting device plugin registration server" Sep 10 15:27:09.772602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772591 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 15:27:09.772653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772606 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 15:27:09.772743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772727 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 15:27:09.772816 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772803 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 15:27:09.772816 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.772815 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 15:27:09.773396 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.773379 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 15:27:09.773478 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.773432 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:09.779739 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.779715 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 15:27:09.780950 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.780934 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 15:27:09.781048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.780984 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 15:27:09.781048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.781004 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 15:27:09.781151 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.781039 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 15:27:09.783013 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.782984 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:09.872939 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.872896 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.874019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.874000 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.874134 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.874031 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.874134 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.874041 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.874134 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.874064 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.881829 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.881810 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal"] Sep 10 15:27:09.881894 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.881873 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.883311 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.883293 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.883407 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.883318 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-82.ec2.internal\": node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:09.883458 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.883436 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.883493 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.883462 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.883493 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.883472 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.884626 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.884614 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.884758 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.884744 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.884807 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.884775 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.885428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.885413 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.885491 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.885437 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.885491 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.885447 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.885955 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.885937 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.886024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.885973 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.886024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.885988 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.887045 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.887031 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.887109 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.887061 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:27:09.888074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.888058 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:27:09.888152 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.888081 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:27:09.888152 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.888096 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:27:09.901023 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.901005 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:09.908299 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.908279 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.ec2.internal\" not found" node="ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.912119 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:09.912102 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.ec2.internal\" not found" node="ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.980379 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.980354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b2e6a28660fe6b5bb6afb57b77aab26-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.ec2.internal\" (UID: \"6b2e6a28660fe6b5bb6afb57b77aab26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.980491 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.980381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09fc35dc40fb211550c2a0c02d259ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"09fc35dc40fb211550c2a0c02d259ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:09.980491 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:09.980402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09fc35dc40fb211550c2a0c02d259ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"09fc35dc40fb211550c2a0c02d259ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.001619 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:10.001566 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:10.081319 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.081290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09fc35dc40fb211550c2a0c02d259ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"09fc35dc40fb211550c2a0c02d259ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.081419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.081323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09fc35dc40fb211550c2a0c02d259ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"09fc35dc40fb211550c2a0c02d259ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.081419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.081343 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b2e6a28660fe6b5bb6afb57b77aab26-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.ec2.internal\" (UID: \"6b2e6a28660fe6b5bb6afb57b77aab26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.081419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.081387 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b2e6a28660fe6b5bb6afb57b77aab26-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.ec2.internal\" (UID: \"6b2e6a28660fe6b5bb6afb57b77aab26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.081419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.081393 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09fc35dc40fb211550c2a0c02d259ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"09fc35dc40fb211550c2a0c02d259ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.081419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.081393 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09fc35dc40fb211550c2a0c02d259ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal\" (UID: \"09fc35dc40fb211550c2a0c02d259ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.102406 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:10.102384 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:10.202948 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:10.202913 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:10.211089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.211066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.214977 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:10.214964 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" Sep 10 15:27:10.303823 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:10.303793 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:11.404592 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:11.404565 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:11.473073 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.473044 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:11.505426 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:11.505393 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:11.510546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.510530 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 15:27:11.510676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.510655 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:27:11.510713 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.510695 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:27:11.510748 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.510695 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:27:11.577674 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.577605 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:22:10 +0000 UTC" deadline="2028-02-08 12:55:06.335874331 +0000 UTC" Sep 10 15:27:11.577674 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.577638 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12381h27m54.758242935s" Sep 10 15:27:11.606169 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:11.606136 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:11.706654 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:11.706622 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:11.719901 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:11.719853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2e6a28660fe6b5bb6afb57b77aab26.slice/crio-b57560fe0db564aba78e6ebe9c4cf5828b48dd2897fc793a6885be3759ee13e2 WatchSource:0}: Error finding container b57560fe0db564aba78e6ebe9c4cf5828b48dd2897fc793a6885be3759ee13e2: Status 404 returned error can't find the container with id b57560fe0db564aba78e6ebe9c4cf5828b48dd2897fc793a6885be3759ee13e2 Sep 10 15:27:11.720473 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:11.720448 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fc35dc40fb211550c2a0c02d259ed5.slice/crio-65a70dee4abbbe967a3c66c2917cad66531536967a43a3648457a6ce8007746c WatchSource:0}: Error finding container 65a70dee4abbbe967a3c66c2917cad66531536967a43a3648457a6ce8007746c: Status 404 returned error can't find the container with id 65a70dee4abbbe967a3c66c2917cad66531536967a43a3648457a6ce8007746c Sep 10 15:27:11.724458 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.724437 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:27:11.786916 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.786862 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" event={"ID":"6b2e6a28660fe6b5bb6afb57b77aab26","Type":"ContainerStarted","Data":"b57560fe0db564aba78e6ebe9c4cf5828b48dd2897fc793a6885be3759ee13e2"} Sep 10 15:27:11.788042 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:11.788020 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" event={"ID":"09fc35dc40fb211550c2a0c02d259ed5","Type":"ContainerStarted","Data":"65a70dee4abbbe967a3c66c2917cad66531536967a43a3648457a6ce8007746c"} Sep 10 15:27:11.807254 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:11.807224 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:11.907814 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:11.907717 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:12.008257 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.008223 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-82.ec2.internal\" not found" Sep 10 15:27:12.029465 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.029431 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:12.077677 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.077650 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" Sep 10 15:27:12.090361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.090336 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:27:12.091533 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.091521 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" Sep 10 15:27:12.101842 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.101825 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:27:12.390423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.390393 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:12.559385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.559351 2581 apiserver.go:52] "Watching apiserver" Sep 10 15:27:12.565552 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.565527 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 15:27:12.566888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.566863 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-6zgzh","openshift-ovn-kubernetes/ovnkube-node-hkm4r","kube-system/konnectivity-agent-vwd5v","kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7","openshift-cluster-node-tuning-operator/tuned-csxb5","openshift-dns/node-resolver-pgfdg","openshift-image-registry/node-ca-r5ph8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal","openshift-multus/network-metrics-daemon-9rlrq","kube-system/global-pull-secret-syncer-4wc8d","openshift-multus/multus-225gp","openshift-multus/multus-additional-cni-plugins-wd2xg","openshift-network-diagnostics/network-check-target-q5rkh"] Sep 10 15:27:12.568291 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.568257 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-225gp" Sep 10 15:27:12.569895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.569875 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.570568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.570548 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 15:27:12.570756 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.570727 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.570854 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.570782 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hlkng\"" Sep 10 15:27:12.570975 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.570958 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.571122 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.571100 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 15:27:12.571842 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.571826 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.572414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.572387 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 15:27:12.572552 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.572533 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 15:27:12.572669 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.572653 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.572900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.572884 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vx8kw\"" Sep 10 15:27:12.573418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.573188 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.573418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.573243 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 15:27:12.573418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.573372 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 15:27:12.573610 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.573519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:12.573610 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.573582 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:12.573733 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.573717 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ww5jn\"" Sep 10 15:27:12.574153 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.574131 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 15:27:12.574237 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.574174 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 15:27:12.574572 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.574555 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.575829 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.575810 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.577061 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.576648 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 15:27:12.577061 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.576790 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.577061 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.576879 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.577061 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.576948 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kj8n2\"" Sep 10 15:27:12.577304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.577187 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.577786 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.577755 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 15:27:12.577972 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.577953 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.578121 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.578104 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2njt\"" Sep 10 15:27:12.578319 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.578302 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.578617 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.578556 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.580901 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.581009 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v978l\"" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.581048 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.581519 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.581567 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.581519 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6cbtf\"" Sep 10 15:27:12.581726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.581700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.582152 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.582133 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.582513 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.582496 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.583009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.582836 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:12.583009 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.582950 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:12.584241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.584220 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 15:27:12.584714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.584581 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 15:27:12.584830 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.584810 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pl4xd\"" Sep 10 15:27:12.585581 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.585389 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.586745 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.586728 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.586885 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.586866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:12.587379 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.587361 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 15:27:12.587625 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.587365 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 15:27:12.587745 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.587421 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q6dsw\"" Sep 10 15:27:12.589890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.589854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-sys-fs\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.589962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.589904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-host-slash\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.589962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.589927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f094e431-2b4b-42cb-baa5-f9abcf436975-cni-binary-copy\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.589962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.589951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-cni-multus\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.589976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-node-log\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:12.590115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-netns\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590097 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-daemon-config\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-systemd\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-systemd\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.590258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-conf-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5p4\" (UniqueName: \"kubernetes.io/projected/f094e431-2b4b-42cb-baa5-f9abcf436975-kube-api-access-jx5p4\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-etc-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-cni-netd\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovn-node-metrics-cert\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590510 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590368 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-run\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.590510 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.590510 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-log-socket\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovnkube-script-lib\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d7cb1dfd-6226-4ca4-be73-b1396fe39b5e-agent-certs\") pod \"konnectivity-agent-vwd5v\" (UID: \"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e\") " pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.590665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-sys\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.590665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590582 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-device-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.590665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-iptables-alerter-script\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.590665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5p7v\" (UniqueName: \"kubernetes.io/projected/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-kube-api-access-w5p7v\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-socket-dir-parent\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-systemd-units\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovnkube-config\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590741 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-kubernetes\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590794 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-var-lib-kubelet\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-kubelet\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590839 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-cni-bin\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590863 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvqk\" (UniqueName: \"kubernetes.io/projected/6a58e490-0c24-42ee-9858-723f04f077c3-kube-api-access-5hvqk\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-cnibin\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590906 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-k8s-cni-cncf-io\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.590937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-ovn\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-env-overrides\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.590978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysctl-conf\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-kubelet\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb1dfd-6226-4ca4-be73-b1396fe39b5e-konnectivity-ca\") pod \"konnectivity-agent-vwd5v\" (UID: \"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e\") " pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-tuned\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-cni-bin\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-host\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmffv\" (UniqueName: \"kubernetes.io/projected/6cbb049d-4320-4bb3-8f6d-0deda397fd56-kube-api-access-nmffv\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591222 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-multus-certs\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysconfig\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lms\" (UniqueName: \"kubernetes.io/projected/d6c0e308-6a54-40b8-b4c5-a415c95047e7-kube-api-access-k7lms\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cbb049d-4320-4bb3-8f6d-0deda397fd56-serviceca\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-system-cni-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591379 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-etc-kubernetes\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.591480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591404 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysctl-d\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-socket-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-hostroot\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-slash\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-var-lib-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591565 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqpw\" (UniqueName: \"kubernetes.io/projected/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-kube-api-access-vqqpw\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-modprobe-d\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c0e308-6a54-40b8-b4c5-a415c95047e7-tmp\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cbb049d-4320-4bb3-8f6d-0deda397fd56-host\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-cni-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-run-netns\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-lib-modules\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.592250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.591742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-registration-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.592812 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.592336 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:27:12.620906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.620881 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjsfz" Sep 10 15:27:12.633659 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.633637 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjsfz" Sep 10 15:27:12.679309 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.679248 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 15:27:12.692242 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-systemd\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmh9\" (UniqueName: \"kubernetes.io/projected/e14e464b-2049-45bb-b740-7be313711c2c-kube-api-access-wjmh9\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-conf-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5p4\" (UniqueName: \"kubernetes.io/projected/f094e431-2b4b-42cb-baa5-f9abcf436975-kube-api-access-jx5p4\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-etc-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-cni-netd\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692357 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-conf-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovn-node-metrics-cert\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-run\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-log-socket\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692443 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-etc-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692446 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-cni-netd\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovnkube-script-lib\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692481 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-run\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d7cb1dfd-6226-4ca4-be73-b1396fe39b5e-agent-certs\") pod \"konnectivity-agent-vwd5v\" (UID: \"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e\") " pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692544 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-log-socket\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692546 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-sys\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692361 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-systemd\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-device-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-iptables-alerter-script\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692664 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-sys\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5p7v\" (UniqueName: \"kubernetes.io/projected/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-kube-api-access-w5p7v\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-socket-dir-parent\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.692801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-systemd-units\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692732 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-device-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.692793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-socket-dir-parent\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693248 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-systemd-units\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovnkube-config\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-kubernetes\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693336 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693360 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-iptables-alerter-script\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-var-lib-kubelet\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-cnibin\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693425 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-kubelet-config\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-kubelet\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-cni-bin\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvqk\" (UniqueName: \"kubernetes.io/projected/6a58e490-0c24-42ee-9858-723f04f077c3-kube-api-access-5hvqk\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693513 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-kubernetes\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-cnibin\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.693566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-k8s-cni-cncf-io\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-ovn\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-k8s-cni-cncf-io\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-env-overrides\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693649 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-ovn\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysctl-conf\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-kubelet\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693750 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysctl-conf\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb1dfd-6226-4ca4-be73-b1396fe39b5e-konnectivity-ca\") pod \"konnectivity-agent-vwd5v\" (UID: \"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e\") " pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-tuned\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-var-lib-kubelet\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693814 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn7gn\" (UniqueName: \"kubernetes.io/projected/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-kube-api-access-sn7gn\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693840 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-kubelet\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ea8ed65-1c50-40b1-8a47-ade9dd837594-hosts-file\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693868 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-kubelet\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.694360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-cni-bin\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693887 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-cni-bin\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693958 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-cnibin\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693964 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-cni-bin\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.693991 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea8ed65-1c50-40b1-8a47-ade9dd837594-tmp-dir\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694013 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-host\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmffv\" (UniqueName: \"kubernetes.io/projected/6cbb049d-4320-4bb3-8f6d-0deda397fd56-kube-api-access-nmffv\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694106 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-host\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-multus-certs\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysconfig\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694237 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-multus-certs\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694275 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lms\" (UniqueName: \"kubernetes.io/projected/d6c0e308-6a54-40b8-b4c5-a415c95047e7-kube-api-access-k7lms\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694308 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb1dfd-6226-4ca4-be73-b1396fe39b5e-konnectivity-ca\") pod \"konnectivity-agent-vwd5v\" (UID: \"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e\") " pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694316 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysconfig\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cbb049d-4320-4bb3-8f6d-0deda397fd56-serviceca\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-system-cni-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-etc-kubernetes\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694465 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysctl-d\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-system-cni-dir\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694508 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-etc-kubernetes\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-socket-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694581 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694660 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-sysctl-d\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-hostroot\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-socket-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-slash\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694799 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-slash\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.695888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-var-lib-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqpw\" (UniqueName: \"kubernetes.io/projected/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-kube-api-access-vqqpw\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-hostroot\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-env-overrides\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-modprobe-d\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c0e308-6a54-40b8-b4c5-a415c95047e7-tmp\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694944 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cbb049d-4320-4bb3-8f6d-0deda397fd56-serviceca\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cbb049d-4320-4bb3-8f6d-0deda397fd56-host\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.694986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-modprobe-d\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695016 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-system-cni-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695039 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cbb049d-4320-4bb3-8f6d-0deda397fd56-host\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695057 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-var-lib-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-dbus\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-cni-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-run-netns\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695252 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-host-run-netns\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-lib-modules\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-registration-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.696722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695300 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-cni-dir\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovnkube-script-lib\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-sys-fs\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695375 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-registration-dir\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6c0e308-6a54-40b8-b4c5-a415c95047e7-lib-modules\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-host-slash\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695417 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a58e490-0c24-42ee-9858-723f04f077c3-sys-fs\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpr9\" (UniqueName: \"kubernetes.io/projected/1ea8ed65-1c50-40b1-8a47-ade9dd837594-kube-api-access-qnpr9\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f094e431-2b4b-42cb-baa5-f9abcf436975-cni-binary-copy\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-cni-multus\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695556 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-node-log\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695607 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-netns\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-daemon-config\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-systemd\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695734 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-systemd\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.697546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-var-lib-cni-multus\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695816 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-run-openvswitch\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-node-log\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.695880 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovnkube-config\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.696054 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f094e431-2b4b-42cb-baa5-f9abcf436975-host-run-netns\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.696099 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-host-slash\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.696496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f094e431-2b4b-42cb-baa5-f9abcf436975-cni-binary-copy\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.697324 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f094e431-2b4b-42cb-baa5-f9abcf436975-multus-daemon-config\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.697706 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-ovn-node-metrics-cert\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.697740 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6c0e308-6a54-40b8-b4c5-a415c95047e7-etc-tuned\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.698346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.698035 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d7cb1dfd-6226-4ca4-be73-b1396fe39b5e-agent-certs\") pod \"konnectivity-agent-vwd5v\" (UID: \"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e\") " pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.701188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.701167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c0e308-6a54-40b8-b4c5-a415c95047e7-tmp\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.703454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.703423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvqk\" (UniqueName: \"kubernetes.io/projected/6a58e490-0c24-42ee-9858-723f04f077c3-kube-api-access-5hvqk\") pod \"aws-ebs-csi-driver-node-44qj7\" (UID: \"6a58e490-0c24-42ee-9858-723f04f077c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.704050 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.704015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5p7v\" (UniqueName: \"kubernetes.io/projected/3c4e5cbf-3105-4edb-805b-19ea38cf5c57-kube-api-access-w5p7v\") pod \"iptables-alerter-6zgzh\" (UID: \"3c4e5cbf-3105-4edb-805b-19ea38cf5c57\") " pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.704181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.704064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmffv\" (UniqueName: \"kubernetes.io/projected/6cbb049d-4320-4bb3-8f6d-0deda397fd56-kube-api-access-nmffv\") pod \"node-ca-r5ph8\" (UID: \"6cbb049d-4320-4bb3-8f6d-0deda397fd56\") " pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.704900 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.704880 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:12.705016 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.704904 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:12.705016 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.704917 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:12.705016 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.704981 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:27:13.204961795 +0000 UTC m=+2.988174855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:12.705926 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.705907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5p4\" (UniqueName: \"kubernetes.io/projected/f094e431-2b4b-42cb-baa5-f9abcf436975-kube-api-access-jx5p4\") pod \"multus-225gp\" (UID: \"f094e431-2b4b-42cb-baa5-f9abcf436975\") " pod="openshift-multus/multus-225gp" Sep 10 15:27:12.706088 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.706071 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lms\" (UniqueName: \"kubernetes.io/projected/d6c0e308-6a54-40b8-b4c5-a415c95047e7-kube-api-access-k7lms\") pod \"tuned-csxb5\" (UID: \"d6c0e308-6a54-40b8-b4c5-a415c95047e7\") " pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.708057 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.708037 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqpw\" (UniqueName: \"kubernetes.io/projected/b8629c0b-9a49-4e22-a6c6-25a6e6d6de29-kube-api-access-vqqpw\") pod \"ovnkube-node-hkm4r\" (UID: \"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.796768 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.796736 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.796768 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.796768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:12.796965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.796795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn7gn\" (UniqueName: \"kubernetes.io/projected/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-kube-api-access-sn7gn\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:12.796965 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.796906 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:12.796965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.796914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ea8ed65-1c50-40b1-8a47-ade9dd837594-hosts-file\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.796965 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.796960 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:13.296942142 +0000 UTC m=+3.080155184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.796978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea8ed65-1c50-40b1-8a47-ade9dd837594-tmp-dir\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797004 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ea8ed65-1c50-40b1-8a47-ade9dd837594-hosts-file\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.796988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-system-cni-dir\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797154 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-system-cni-dir\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-dbus\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpr9\" (UniqueName: \"kubernetes.io/projected/1ea8ed65-1c50-40b1-8a47-ade9dd837594-kube-api-access-qnpr9\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797291 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea8ed65-1c50-40b1-8a47-ade9dd837594-tmp-dir\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797302 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-dbus\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.797333 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmh9\" (UniqueName: \"kubernetes.io/projected/e14e464b-2049-45bb-b740-7be313711c2c-kube-api-access-wjmh9\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:12.797399 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:13.297381468 +0000 UTC m=+3.080594536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-cnibin\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.797548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-kubelet-config\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.798009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e14e464b-2049-45bb-b740-7be313711c2c-cnibin\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.798009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-kubelet-config\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:12.798009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797636 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.798009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.798009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.797916 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e14e464b-2049-45bb-b740-7be313711c2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.810058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.810034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmh9\" (UniqueName: \"kubernetes.io/projected/e14e464b-2049-45bb-b740-7be313711c2c-kube-api-access-wjmh9\") pod \"multus-additional-cni-plugins-wd2xg\" (UID: \"e14e464b-2049-45bb-b740-7be313711c2c\") " pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.810321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.810295 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpr9\" (UniqueName: \"kubernetes.io/projected/1ea8ed65-1c50-40b1-8a47-ade9dd837594-kube-api-access-qnpr9\") pod \"node-resolver-pgfdg\" (UID: \"1ea8ed65-1c50-40b1-8a47-ade9dd837594\") " pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:12.810578 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.810553 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn7gn\" (UniqueName: \"kubernetes.io/projected/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-kube-api-access-sn7gn\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:12.859371 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.859341 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:12.882747 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.882719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-225gp" Sep 10 15:27:12.890018 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.889981 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:12.898891 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.898864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:12.905674 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.905655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" Sep 10 15:27:12.912347 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.912319 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-csxb5" Sep 10 15:27:12.920857 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.920837 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6zgzh" Sep 10 15:27:12.930149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.930103 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5ph8" Sep 10 15:27:12.935581 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.935563 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" Sep 10 15:27:12.940163 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:12.940146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgfdg" Sep 10 15:27:13.300855 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.300822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:13.301016 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.300864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:13.301016 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.300893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:13.301016 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.300970 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:13.301016 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.300986 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:13.301016 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.301018 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:13.301198 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.301029 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:13.301198 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.301034 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:14.30102019 +0000 UTC m=+4.084233246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:13.301198 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.300971 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:13.301198 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.301070 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:27:14.301058017 +0000 UTC m=+4.084271083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:13.301198 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.301087 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:14.301075889 +0000 UTC m=+4.084288931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:13.597327 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.597131 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cb1dfd_6226_4ca4_be73_b1396fe39b5e.slice/crio-16e1f80399a6b7eb59a564b70889a839e6ceb85106dfe95675099f3c0f26808f WatchSource:0}: Error finding container 16e1f80399a6b7eb59a564b70889a839e6ceb85106dfe95675099f3c0f26808f: Status 404 returned error can't find the container with id 16e1f80399a6b7eb59a564b70889a839e6ceb85106dfe95675099f3c0f26808f Sep 10 15:27:13.600936 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.600911 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4e5cbf_3105_4edb_805b_19ea38cf5c57.slice/crio-aaeca73dcd90bbebac16036a0fffd8b634a25b98e261ebe742ad0996c51a03da WatchSource:0}: Error finding container aaeca73dcd90bbebac16036a0fffd8b634a25b98e261ebe742ad0996c51a03da: Status 404 returned error can't find the container with id aaeca73dcd90bbebac16036a0fffd8b634a25b98e261ebe742ad0996c51a03da Sep 10 15:27:13.601669 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.601648 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea8ed65_1c50_40b1_8a47_ade9dd837594.slice/crio-46f218d46f03fc5e041af4d3584b012b0e2df9fef28cabc6b440832220752ed0 WatchSource:0}: Error finding container 46f218d46f03fc5e041af4d3584b012b0e2df9fef28cabc6b440832220752ed0: Status 404 returned error can't find the container with id 46f218d46f03fc5e041af4d3584b012b0e2df9fef28cabc6b440832220752ed0 Sep 10 15:27:13.602264 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.602234 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8629c0b_9a49_4e22_a6c6_25a6e6d6de29.slice/crio-f20b060f7d65cdb714a80bac23c6fe87fc4d1cafaa7b09678ea1f54bb316cd4a WatchSource:0}: Error finding container f20b060f7d65cdb714a80bac23c6fe87fc4d1cafaa7b09678ea1f54bb316cd4a: Status 404 returned error can't find the container with id f20b060f7d65cdb714a80bac23c6fe87fc4d1cafaa7b09678ea1f54bb316cd4a Sep 10 15:27:13.603536 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.603492 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbb049d_4320_4bb3_8f6d_0deda397fd56.slice/crio-b5d3cdc4694949e7a9650f7373d90998f3c33d06de1626ecaf89e0040d781ec4 WatchSource:0}: Error finding container b5d3cdc4694949e7a9650f7373d90998f3c33d06de1626ecaf89e0040d781ec4: Status 404 returned error can't find the container with id b5d3cdc4694949e7a9650f7373d90998f3c33d06de1626ecaf89e0040d781ec4 Sep 10 15:27:13.604344 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.604191 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a58e490_0c24_42ee_9858_723f04f077c3.slice/crio-fd58cf504e585abef934112ef55ed8cae6026acff7c9345974ae3e21f5b237d0 WatchSource:0}: Error finding container fd58cf504e585abef934112ef55ed8cae6026acff7c9345974ae3e21f5b237d0: Status 404 returned error can't find the container with id fd58cf504e585abef934112ef55ed8cae6026acff7c9345974ae3e21f5b237d0 Sep 10 15:27:13.605295 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:13.605187 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c0e308_6a54_40b8_b4c5_a415c95047e7.slice/crio-8580882b60d8cb9aac1570baeaa4f0dd587805508ebe5ab75952dbf156761028 WatchSource:0}: Error finding container 8580882b60d8cb9aac1570baeaa4f0dd587805508ebe5ab75952dbf156761028: Status 404 returned error can't find the container with id 8580882b60d8cb9aac1570baeaa4f0dd587805508ebe5ab75952dbf156761028 Sep 10 15:27:13.635421 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.635353 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:22:12 +0000 UTC" deadline="2028-03-12 20:28:16.928447364 +0000 UTC" Sep 10 15:27:13.635421 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.635421 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13181h1m3.293030729s" Sep 10 15:27:13.782377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.782341 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:13.782522 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.782358 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:13.782522 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.782460 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:13.782522 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.782484 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:13.782651 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.782604 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:13.782702 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:13.782687 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:13.794789 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.794761 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgfdg" event={"ID":"1ea8ed65-1c50-40b1-8a47-ade9dd837594","Type":"ContainerStarted","Data":"46f218d46f03fc5e041af4d3584b012b0e2df9fef28cabc6b440832220752ed0"} Sep 10 15:27:13.796301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.796276 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-csxb5" event={"ID":"d6c0e308-6a54-40b8-b4c5-a415c95047e7","Type":"ContainerStarted","Data":"8580882b60d8cb9aac1570baeaa4f0dd587805508ebe5ab75952dbf156761028"} Sep 10 15:27:13.798536 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.798509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" event={"ID":"6a58e490-0c24-42ee-9858-723f04f077c3","Type":"ContainerStarted","Data":"fd58cf504e585abef934112ef55ed8cae6026acff7c9345974ae3e21f5b237d0"} Sep 10 15:27:13.800176 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.800130 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5ph8" event={"ID":"6cbb049d-4320-4bb3-8f6d-0deda397fd56","Type":"ContainerStarted","Data":"b5d3cdc4694949e7a9650f7373d90998f3c33d06de1626ecaf89e0040d781ec4"} Sep 10 15:27:13.801345 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.801323 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"f20b060f7d65cdb714a80bac23c6fe87fc4d1cafaa7b09678ea1f54bb316cd4a"} Sep 10 15:27:13.803252 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.803230 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" event={"ID":"6b2e6a28660fe6b5bb6afb57b77aab26","Type":"ContainerStarted","Data":"6899d75af8407c49aee4b89b12ba9c81e59e032964b665a1f62572e03a187134"} Sep 10 15:27:13.804393 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.804370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerStarted","Data":"d94bc8fdda3fa6ebfac38a2427354313ff358cd9bd846ebb189227422636d945"} Sep 10 15:27:13.805565 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.805540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-225gp" event={"ID":"f094e431-2b4b-42cb-baa5-f9abcf436975","Type":"ContainerStarted","Data":"3ed91098472a062858fe75622df9db115f46d487c931e5ff4166b137ff4dc1cb"} Sep 10 15:27:13.806653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.806633 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6zgzh" event={"ID":"3c4e5cbf-3105-4edb-805b-19ea38cf5c57","Type":"ContainerStarted","Data":"aaeca73dcd90bbebac16036a0fffd8b634a25b98e261ebe742ad0996c51a03da"} Sep 10 15:27:13.807749 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.807730 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwd5v" event={"ID":"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e","Type":"ContainerStarted","Data":"16e1f80399a6b7eb59a564b70889a839e6ceb85106dfe95675099f3c0f26808f"} Sep 10 15:27:13.818477 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:13.818429 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.ec2.internal" podStartSLOduration=1.818418174 podStartE2EDuration="1.818418174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:13.81828365 +0000 UTC m=+3.601496714" watchObservedRunningTime="2026-09-10 15:27:13.818418174 +0000 UTC m=+3.601631238" Sep 10 15:27:14.158954 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.158878 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:27:14.307956 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.307919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:14.308133 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308084 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:14.308133 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308120 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:14.308244 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.307990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:14.308244 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308159 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:16.308141194 +0000 UTC m=+6.091354242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:14.308244 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308196 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:16.308177474 +0000 UTC m=+6.091390526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:14.308400 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.308257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:14.308458 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308423 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:14.308458 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308440 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:14.308458 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308452 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:14.308595 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:14.308491 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:27:16.308478557 +0000 UTC m=+6.091691603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:14.635740 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.635693 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:22:12 +0000 UTC" deadline="2028-04-17 07:04:08.235887478 +0000 UTC" Sep 10 15:27:14.635740 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.635736 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14031h36m53.600155551s" Sep 10 15:27:14.841139 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.841095 2581 generic.go:356] "Generic (PLEG): container finished" podID="09fc35dc40fb211550c2a0c02d259ed5" containerID="e24556cbb4d5cc4d14dad51a8332c67a7c0c3e3dd0056383c1a2be9e58ea16fb" exitCode=0 Sep 10 15:27:14.841842 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:14.841347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" event={"ID":"09fc35dc40fb211550c2a0c02d259ed5","Type":"ContainerDied","Data":"e24556cbb4d5cc4d14dad51a8332c67a7c0c3e3dd0056383c1a2be9e58ea16fb"} Sep 10 15:27:15.781724 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:15.781694 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:15.782315 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:15.781821 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:15.782315 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:15.782214 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:15.782418 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:15.782348 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:15.782418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:15.782400 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:15.782528 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:15.782491 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:15.855142 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:15.855106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" event={"ID":"09fc35dc40fb211550c2a0c02d259ed5","Type":"ContainerStarted","Data":"7e25478a1cb2c30cc466e720bb1baf649935613f43fd805be78d3584b806f83b"} Sep 10 15:27:16.326752 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:16.326714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:16.326928 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:16.326778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:16.326928 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:16.326818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:16.327083 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.326960 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:16.327083 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.326979 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:16.327083 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.327006 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:16.327083 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.327071 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:27:20.327049984 +0000 UTC m=+10.110263095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:16.327277 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.327187 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:16.327277 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.327199 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:16.327277 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.327235 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:20.327221734 +0000 UTC m=+10.110434784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:16.327277 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:16.327265 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:20.327246691 +0000 UTC m=+10.110459746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:17.781893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:17.781859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:17.782349 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:17.782010 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:17.782439 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:17.782402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:17.782513 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:17.782491 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:17.782568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:17.782554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:17.782651 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:17.782632 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:19.782231 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:19.782155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:19.782678 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:19.782277 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:19.782678 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:19.782368 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:19.782678 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:19.782273 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:19.782678 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:19.782418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:19.782678 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:19.782479 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:20.358987 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:20.358949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:20.359195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:20.359030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:20.359195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:20.359077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:20.359195 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359112 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:20.359195 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359173 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:20.359398 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359263 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:20.359398 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359276 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:20.359398 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359288 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:20.359398 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359175 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:28.359157033 +0000 UTC m=+18.142370092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:20.359398 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359329 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:28.359317118 +0000 UTC m=+18.142530166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:20.359398 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:20.359343 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:27:28.359334344 +0000 UTC m=+18.142547387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:21.781671 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:21.781591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:21.781671 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:21.781630 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:21.782195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:21.781591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:21.782195 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:21.781733 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:21.782195 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:21.781784 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:21.782195 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:21.781847 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:23.781642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:23.781603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:23.781642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:23.781628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:23.781642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:23.781641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:23.782181 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:23.781744 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:23.782181 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:23.781858 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:23.782181 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:23.781930 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:25.782439 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:25.782217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:25.782819 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:25.782227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:25.782819 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:25.782523 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:25.782819 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:25.782270 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:25.782819 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:25.782586 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:25.782819 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:25.782691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:27.781868 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:27.781827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:27.782422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:27.781841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:27.782422 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:27.781970 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:27.782422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:27.782014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:27.782422 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:27.782093 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:27.782422 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:27.782176 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:28.415735 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:28.415697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:28.415919 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:28.415757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:28.415919 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:28.415812 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:28.415919 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.415861 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:28.415919 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.415915 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:28.416130 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.415943 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:44.415919852 +0000 UTC m=+34.199132895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:28.416130 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.415952 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:28.416130 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.415967 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:44.415951144 +0000 UTC m=+34.199164206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:28.416130 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.415979 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:28.416130 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.416006 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:28.416130 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:28.416058 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:27:44.416043835 +0000 UTC m=+34.199256880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:29.781752 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:29.781717 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:29.781752 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:29.781748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:29.782190 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:29.781755 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:29.782190 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:29.781825 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:29.782190 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:29.781909 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:29.782190 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:29.782013 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:31.784372 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.784160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:31.785070 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.784179 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:31.785070 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:31.784473 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:31.785070 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.784185 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:31.785070 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:31.784528 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:31.785070 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:31.784572 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:31.888657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.888617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" event={"ID":"6a58e490-0c24-42ee-9858-723f04f077c3","Type":"ContainerStarted","Data":"457e7dbd2ef8434c7da48f7f3825597b55fa74ad59c698f0f82bb4c83ad96435"} Sep 10 15:27:31.890386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.890355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5ph8" event={"ID":"6cbb049d-4320-4bb3-8f6d-0deda397fd56","Type":"ContainerStarted","Data":"7990edb77f971cd3e2ac74a71bd0c8940109646d9dd1f242b5d92b961cc4d11d"} Sep 10 15:27:31.894644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.894512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"79262da66980e0b0837d8e527ecab6d82bc7c6cf3d2a396a8724cb7dea728cc1"} Sep 10 15:27:31.894644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.894594 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"23e37067688fec3a37178f869bad89452e40ad15a61895851f7d8a75cec6faa0"} Sep 10 15:27:31.894644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.894643 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"afc48f9ee1177763c3b39678db3988811b43703d8f5750de132308da0dbcb139"} Sep 10 15:27:31.894898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.894689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"b3658d710f726459112589f8ed24d827abf0aaa2f58e4d82e02b9b3281beab23"} Sep 10 15:27:31.894898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.894733 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"4535f015d40b079db6f23d3b5995d05a3282bfc7bae0b75cba710cb469dcf0ed"} Sep 10 15:27:31.894898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.894774 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"29300bb9b38662c1e9b3e1003df8e0a7f4ca4eff183026fcad36626d78d830b9"} Sep 10 15:27:31.896619 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.896589 2581 generic.go:356] "Generic (PLEG): container finished" podID="e14e464b-2049-45bb-b740-7be313711c2c" containerID="e132928bf563a6e5710784aa6ff8ef75300d7b8e93b4f565e40b6f51fcbf3992" exitCode=0 Sep 10 15:27:31.896725 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.896676 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerDied","Data":"e132928bf563a6e5710784aa6ff8ef75300d7b8e93b4f565e40b6f51fcbf3992"} Sep 10 15:27:31.898578 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.898552 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-225gp" event={"ID":"f094e431-2b4b-42cb-baa5-f9abcf436975","Type":"ContainerStarted","Data":"42d327ed95d439db4a73d9ec1cedc302f2a3e0c679bc59e2b06bb89b3a983b3a"} Sep 10 15:27:31.900532 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.900509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwd5v" event={"ID":"d7cb1dfd-6226-4ca4-be73-b1396fe39b5e","Type":"ContainerStarted","Data":"f290b879f3ce395d17f00259c4d0b3fdece3456add85a4bcf04f31ffa4856343"} Sep 10 15:27:31.902262 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.902237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgfdg" event={"ID":"1ea8ed65-1c50-40b1-8a47-ade9dd837594","Type":"ContainerStarted","Data":"b3c1f7d1c1257ad6da80fde60bcaa90d34559841da7b9d75a9a752a3585aa2e4"} Sep 10 15:27:31.904096 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.904072 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-csxb5" event={"ID":"d6c0e308-6a54-40b8-b4c5-a415c95047e7","Type":"ContainerStarted","Data":"72dccb6b57fb62d325f021698fb4d0e4d11e728504497b6fc344d0d735248408"} Sep 10 15:27:31.909148 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.909105 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r5ph8" podStartSLOduration=4.459122268 podStartE2EDuration="21.909089621s" totalImagesPullingTime="17.449967353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:31.908054861 +0000 UTC m=+21.691267924" watchObservedRunningTime="2026-09-10 15:27:31.909089621 +0000 UTC m=+21.692302685" Sep 10 15:27:31.909252 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.909198 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.ec2.internal" podStartSLOduration=19.909188763 podStartE2EDuration="19.909188763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:15.877133212 +0000 UTC m=+5.660346277" watchObservedRunningTime="2026-09-10 15:27:31.909188763 +0000 UTC m=+21.692401827" Sep 10 15:27:31.925542 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.925498 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pgfdg" podStartSLOduration=3.472485254 podStartE2EDuration="20.92548555s" totalImagesPullingTime="17.453000296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:31.924964233 +0000 UTC m=+21.708177300" watchObservedRunningTime="2026-09-10 15:27:31.92548555 +0000 UTC m=+21.708698613" Sep 10 15:27:31.948816 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.948765 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vwd5v" podStartSLOduration=8.557390711 podStartE2EDuration="21.948745037s" totalImagesPullingTime="13.391354326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:31.948414087 +0000 UTC m=+21.731627152" watchObservedRunningTime="2026-09-10 15:27:31.948745037 +0000 UTC m=+21.731958102" Sep 10 15:27:31.992746 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:31.992692 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-225gp" podStartSLOduration=4.366358531 podStartE2EDuration="21.992677302s" totalImagesPullingTime="17.626318771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:31.992645178 +0000 UTC m=+21.775858242" watchObservedRunningTime="2026-09-10 15:27:31.992677302 +0000 UTC m=+21.775890366" Sep 10 15:27:32.012361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.012285 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-csxb5" podStartSLOduration=4.568069967 podStartE2EDuration="22.012263959s" totalImagesPullingTime="17.444193992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:32.011978989 +0000 UTC m=+21.795192063" watchObservedRunningTime="2026-09-10 15:27:32.012263959 +0000 UTC m=+21.795477024" Sep 10 15:27:32.586470 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.586445 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 15:27:32.607688 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.607483 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:32.607968 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.607955 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:32.789027 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.788863 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:27:32.58646618Z","UUID":"63f37c6f-3a99-4464-988b-7f2aad24557f","Handler":null,"Name":"","Endpoint":""} Sep 10 15:27:32.790550 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.790531 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 15:27:32.790680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.790559 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 15:27:32.907468 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.907426 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6zgzh" event={"ID":"3c4e5cbf-3105-4edb-805b-19ea38cf5c57","Type":"ContainerStarted","Data":"55bcd7bdc6f8995d2d8f4f74ae001e8de4feb23132ed4d7e211c3f1116924fee"} Sep 10 15:27:32.909564 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.909536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" event={"ID":"6a58e490-0c24-42ee-9858-723f04f077c3","Type":"ContainerStarted","Data":"08fc80f5165b9dae9134333f435efc635dd65a741cb34d1a10264215cfec9fbb"} Sep 10 15:27:32.910494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.910472 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:32.910899 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.910883 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwd5v" Sep 10 15:27:32.942721 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:32.942672 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6zgzh" podStartSLOduration=5.489759411 podStartE2EDuration="22.942657791s" totalImagesPullingTime="17.45289838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:32.924121906 +0000 UTC m=+22.707334969" watchObservedRunningTime="2026-09-10 15:27:32.942657791 +0000 UTC m=+22.725870854" Sep 10 15:27:33.781743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:33.781718 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:33.781846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:33.781814 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:33.781846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:33.781837 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:33.781920 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:33.781825 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:33.781920 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:33.781890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:33.781988 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:33.781942 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:33.915067 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:33.915034 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" event={"ID":"6a58e490-0c24-42ee-9858-723f04f077c3","Type":"ContainerStarted","Data":"e3098e36a346e6ea5ce26fe21f3ef7c922986dca0db2905a117471850197bd04"} Sep 10 15:27:33.933976 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:33.933934 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-44qj7" podStartSLOduration=4.26685158 podStartE2EDuration="23.933917358s" totalImagesPullingTime="19.667065778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:33.933593461 +0000 UTC m=+23.716806525" watchObservedRunningTime="2026-09-10 15:27:33.933917358 +0000 UTC m=+23.717130422" Sep 10 15:27:34.922140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:34.922102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"91f97604635e8673fbd3318f627e77800b251dac291e306e858669dc0ccc16c9"} Sep 10 15:27:35.781586 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:35.781555 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:35.781586 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:35.781573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:35.781586 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:35.781573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:35.781851 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:35.781691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:35.781851 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:35.781815 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:35.781932 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:35.781882 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:36.930617 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:36.930488 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" event={"ID":"b8629c0b-9a49-4e22-a6c6-25a6e6d6de29","Type":"ContainerStarted","Data":"984c7e7ef5a5c040f576a027dffdfb6fb8658371e416c89a9d432b96783d3a1f"} Sep 10 15:27:36.931033 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:36.930682 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:36.945348 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:36.945324 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:36.959981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:36.959680 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" podStartSLOduration=9.479605324 podStartE2EDuration="26.959661735s" totalImagesPullingTime="17.480056411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:36.959500753 +0000 UTC m=+26.742713818" watchObservedRunningTime="2026-09-10 15:27:36.959661735 +0000 UTC m=+26.742874800" Sep 10 15:27:37.782277 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.782198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:37.782408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.782295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:37.782408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.782303 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:37.782408 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:37.782296 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:37.782408 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:37.782371 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:37.782544 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:37.782457 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:37.934488 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.934453 2581 generic.go:356] "Generic (PLEG): container finished" podID="e14e464b-2049-45bb-b740-7be313711c2c" containerID="e7e09672ec42d2e13857a4c4116f2788bc16a41ce09f6269a7079f87512a8f5d" exitCode=0 Sep 10 15:27:37.935058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.934541 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerDied","Data":"e7e09672ec42d2e13857a4c4116f2788bc16a41ce09f6269a7079f87512a8f5d"} Sep 10 15:27:37.935058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.934711 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 15:27:37.935315 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.935220 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:37.949966 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:37.949947 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:38.725277 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.725244 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4wc8d"] Sep 10 15:27:38.725414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.725390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:38.725572 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:38.725481 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:38.728418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.728393 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9rlrq"] Sep 10 15:27:38.728531 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.728495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:38.728594 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:38.728575 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:38.741207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.741144 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5rkh"] Sep 10 15:27:38.741301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.741231 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:38.741335 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:38.741305 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:38.936829 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:38.936801 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 15:27:39.181269 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:39.181116 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:27:39.940985 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:39.940947 2581 generic.go:356] "Generic (PLEG): container finished" podID="e14e464b-2049-45bb-b740-7be313711c2c" containerID="cc2dab094ea0249a76ad00cd9e9d528ad09c010d3784783be2e5ec46f62ab342" exitCode=0 Sep 10 15:27:39.941366 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:39.941031 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerDied","Data":"cc2dab094ea0249a76ad00cd9e9d528ad09c010d3784783be2e5ec46f62ab342"} Sep 10 15:27:40.782536 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:40.782499 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:40.782708 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:40.782602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:40.782708 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:40.782612 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:40.782814 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:40.782735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:40.782814 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:40.782771 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:40.782972 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:40.782816 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:40.945520 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:40.945484 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerStarted","Data":"c2966c5bade2804e084695583d47facbc1bb6dd6834387cba72534f95de89f8f"} Sep 10 15:27:41.949308 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:41.949274 2581 generic.go:356] "Generic (PLEG): container finished" podID="e14e464b-2049-45bb-b740-7be313711c2c" containerID="c2966c5bade2804e084695583d47facbc1bb6dd6834387cba72534f95de89f8f" exitCode=0 Sep 10 15:27:41.949697 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:41.949345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerDied","Data":"c2966c5bade2804e084695583d47facbc1bb6dd6834387cba72534f95de89f8f"} Sep 10 15:27:42.782612 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:42.782574 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:42.782771 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:42.782722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:42.782771 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:42.782724 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:42.782771 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:42.782751 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:42.782931 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:42.782846 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:42.783027 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:42.782966 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:44.425825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.425749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:44.425825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.425789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:44.425825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.425818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425907 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425909 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425919 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425939 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425948 2581 projected.go:196] Error preparing data for projected volume kube-api-access-7qc8l for pod openshift-network-diagnostics/network-check-target-q5rkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425959 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:16.425945471 +0000 UTC m=+66.209158512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425971 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret podName:2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:16.425965216 +0000 UTC m=+66.209178259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret") pod "global-pull-secret-syncer-4wc8d" (UID: "2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:27:44.426285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.425981 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l podName:2b83d159-086f-4d5a-ae13-ba013a2490af nodeName:}" failed. No retries permitted until 2026-09-10 15:28:16.425976204 +0000 UTC m=+66.209189245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qc8l" (UniqueName: "kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l") pod "network-check-target-q5rkh" (UID: "2b83d159-086f-4d5a-ae13-ba013a2490af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:27:44.782548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.782398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:44.782548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.782407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:44.782548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.782525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:44.782548 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.782529 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9rlrq" podUID="e9fb2ac2-2577-4d59-ad17-cb67bd293b36" Sep 10 15:27:44.782856 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.782664 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wc8d" podUID="2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1" Sep 10 15:27:44.782856 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:44.782768 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5rkh" podUID="2b83d159-086f-4d5a-ae13-ba013a2490af" Sep 10 15:27:44.984256 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.984231 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-82.ec2.internal" event="NodeReady" Sep 10 15:27:44.984423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:44.984362 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 15:27:45.042694 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.042618 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wh8rn"] Sep 10 15:27:45.045103 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.045079 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zhxq"] Sep 10 15:27:45.045236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.045170 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.047166 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.047146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.047376 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.047355 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 15:27:45.047691 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.047675 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w5mt4\"" Sep 10 15:27:45.047779 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.047674 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 15:27:45.049273 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.049251 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 15:27:45.049371 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.049305 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 15:27:45.049432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.049385 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8ztg8\"" Sep 10 15:27:45.049605 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.049592 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 15:27:45.054973 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.054955 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zhxq"] Sep 10 15:27:45.057878 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.057840 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wh8rn"] Sep 10 15:27:45.130792 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.130763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/956f0ede-4db9-4494-ad43-b99970036e50-config-volume\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.130965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.130817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.130965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.130852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a1ae1ee-a086-4096-86d3-4ec6ef4672cd-cert\") pod \"ingress-canary-8zhxq\" (UID: \"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd\") " pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.130965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.130889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd7b\" (UniqueName: \"kubernetes.io/projected/3a1ae1ee-a086-4096-86d3-4ec6ef4672cd-kube-api-access-kdd7b\") pod \"ingress-canary-8zhxq\" (UID: \"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd\") " pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.130965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.130915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956f0ede-4db9-4494-ad43-b99970036e50-tmp-dir\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.130965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.130942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg4fx\" (UniqueName: \"kubernetes.io/projected/956f0ede-4db9-4494-ad43-b99970036e50-kube-api-access-hg4fx\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.232084 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956f0ede-4db9-4494-ad43-b99970036e50-tmp-dir\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.232291 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg4fx\" (UniqueName: \"kubernetes.io/projected/956f0ede-4db9-4494-ad43-b99970036e50-kube-api-access-hg4fx\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.232291 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/956f0ede-4db9-4494-ad43-b99970036e50-config-volume\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.232291 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.232456 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a1ae1ee-a086-4096-86d3-4ec6ef4672cd-cert\") pod \"ingress-canary-8zhxq\" (UID: \"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd\") " pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.232456 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:45.232313 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:27:45.232456 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd7b\" (UniqueName: \"kubernetes.io/projected/3a1ae1ee-a086-4096-86d3-4ec6ef4672cd-kube-api-access-kdd7b\") pod \"ingress-canary-8zhxq\" (UID: \"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd\") " pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.232456 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:45.232378 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls podName:956f0ede-4db9-4494-ad43-b99970036e50 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:45.732358166 +0000 UTC m=+35.515571222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls") pod "dns-default-wh8rn" (UID: "956f0ede-4db9-4494-ad43-b99970036e50") : secret "dns-default-metrics-tls" not found Sep 10 15:27:45.232456 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232454 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956f0ede-4db9-4494-ad43-b99970036e50-tmp-dir\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.232941 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.232924 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/956f0ede-4db9-4494-ad43-b99970036e50-config-volume\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.236863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.236842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a1ae1ee-a086-4096-86d3-4ec6ef4672cd-cert\") pod \"ingress-canary-8zhxq\" (UID: \"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd\") " pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.244633 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.244611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg4fx\" (UniqueName: \"kubernetes.io/projected/956f0ede-4db9-4494-ad43-b99970036e50-kube-api-access-hg4fx\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.244761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.244671 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd7b\" (UniqueName: \"kubernetes.io/projected/3a1ae1ee-a086-4096-86d3-4ec6ef4672cd-kube-api-access-kdd7b\") pod \"ingress-canary-8zhxq\" (UID: \"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd\") " pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.362980 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.362940 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zhxq" Sep 10 15:27:45.735395 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:45.735318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:45.735797 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:45.735495 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:27:45.735797 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:45.735577 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls podName:956f0ede-4db9-4494-ad43-b99970036e50 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:46.735554007 +0000 UTC m=+36.518767055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls") pod "dns-default-wh8rn" (UID: "956f0ede-4db9-4494-ad43-b99970036e50") : secret "dns-default-metrics-tls" not found Sep 10 15:27:46.743264 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.743121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:46.743751 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:46.743264 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:27:46.743751 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:46.743324 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls podName:956f0ede-4db9-4494-ad43-b99970036e50 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:48.743308967 +0000 UTC m=+38.526522014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls") pod "dns-default-wh8rn" (UID: "956f0ede-4db9-4494-ad43-b99970036e50") : secret "dns-default-metrics-tls" not found Sep 10 15:27:46.782275 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.782247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:27:46.782423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.782247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:27:46.784115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.782247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:27:46.785162 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.784941 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:27:46.785162 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.784987 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:27:46.786047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.785968 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rkds\"" Sep 10 15:27:46.787500 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.786842 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:27:46.787598 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.787571 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:27:46.791620 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.789418 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8jcj\"" Sep 10 15:27:46.791620 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.790518 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zhxq"] Sep 10 15:27:46.795026 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:46.794974 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1ae1ee_a086_4096_86d3_4ec6ef4672cd.slice/crio-4e86c10a702befac97c3603045685f0dedc6db5f52dd19ade145fcaff1e3ad7c WatchSource:0}: Error finding container 4e86c10a702befac97c3603045685f0dedc6db5f52dd19ade145fcaff1e3ad7c: Status 404 returned error can't find the container with id 4e86c10a702befac97c3603045685f0dedc6db5f52dd19ade145fcaff1e3ad7c Sep 10 15:27:46.891059 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.891029 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-59q64"] Sep 10 15:27:46.902084 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.902066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:46.904429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.904409 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 15:27:46.904539 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.904444 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 15:27:46.904539 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.904478 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 15:27:46.904670 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.904657 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cg9jt\"" Sep 10 15:27:46.904736 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.904708 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 15:27:46.906964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.906947 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-59q64"] Sep 10 15:27:46.909142 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.909119 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 15:27:46.963919 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.963888 2581 generic.go:356] "Generic (PLEG): container finished" podID="e14e464b-2049-45bb-b740-7be313711c2c" containerID="903d7c2b2851df469d6c1fde0230ab2a1ca2ea38c85480d2137644b0df4ff3e6" exitCode=0 Sep 10 15:27:46.964056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.963962 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerDied","Data":"903d7c2b2851df469d6c1fde0230ab2a1ca2ea38c85480d2137644b0df4ff3e6"} Sep 10 15:27:46.965276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.965257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zhxq" event={"ID":"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd","Type":"ContainerStarted","Data":"4e86c10a702befac97c3603045685f0dedc6db5f52dd19ade145fcaff1e3ad7c"} Sep 10 15:27:46.996372 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:46.996347 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7fd6757d9d-prmxg"] Sep 10 15:27:47.006153 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.006129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.008621 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.008567 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 15:27:47.008853 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.008826 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 15:27:47.008927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.008872 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 15:27:47.008927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.008889 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2bpf8\"" Sep 10 15:27:47.009057 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.008936 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 15:27:47.009057 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.008877 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 15:27:47.009242 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.009225 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 15:27:47.015525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.015507 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fd6757d9d-prmxg"] Sep 10 15:27:47.046171 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.046153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qt46\" (UniqueName: \"kubernetes.io/projected/2e1d53ff-0078-41fe-9d4e-66c0629657a1-kube-api-access-8qt46\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.046263 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.046195 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e1d53ff-0078-41fe-9d4e-66c0629657a1-tmp\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.046263 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.046212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e1d53ff-0078-41fe-9d4e-66c0629657a1-serving-cert\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.046421 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.046399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e1d53ff-0078-41fe-9d4e-66c0629657a1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.046630 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.046606 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e1d53ff-0078-41fe-9d4e-66c0629657a1-snapshots\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.046734 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.046640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e1d53ff-0078-41fe-9d4e-66c0629657a1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.147908 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.147880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e1d53ff-0078-41fe-9d4e-66c0629657a1-tmp\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.147915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn47p\" (UniqueName: \"kubernetes.io/projected/16ae4a17-d168-4499-8387-639457cdb1d4-kube-api-access-hn47p\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.148049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.147935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e1d53ff-0078-41fe-9d4e-66c0629657a1-serving-cert\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.147959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-stats-auth\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.148049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e1d53ff-0078-41fe-9d4e-66c0629657a1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148202 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148066 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.148202 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-default-certificate\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.148276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e1d53ff-0078-41fe-9d4e-66c0629657a1-snapshots\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e1d53ff-0078-41fe-9d4e-66c0629657a1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148355 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e1d53ff-0078-41fe-9d4e-66c0629657a1-tmp\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148355 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qt46\" (UniqueName: \"kubernetes.io/projected/2e1d53ff-0078-41fe-9d4e-66c0629657a1-kube-api-access-8qt46\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148355 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.148753 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148734 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e1d53ff-0078-41fe-9d4e-66c0629657a1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148814 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148750 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e1d53ff-0078-41fe-9d4e-66c0629657a1-snapshots\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.148924 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.148908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e1d53ff-0078-41fe-9d4e-66c0629657a1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.150653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.150637 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e1d53ff-0078-41fe-9d4e-66c0629657a1-serving-cert\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.160189 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.160169 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qt46\" (UniqueName: \"kubernetes.io/projected/2e1d53ff-0078-41fe-9d4e-66c0629657a1-kube-api-access-8qt46\") pod \"insights-operator-76bcd6ccf5-59q64\" (UID: \"2e1d53ff-0078-41fe-9d4e-66c0629657a1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.233195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.233149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" Sep 10 15:27:47.249101 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.249078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.249205 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.249115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn47p\" (UniqueName: \"kubernetes.io/projected/16ae4a17-d168-4499-8387-639457cdb1d4-kube-api-access-hn47p\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.249205 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.249136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-stats-auth\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.249205 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.249186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.249359 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.249226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-default-certificate\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.249359 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:47.249251 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:47.749232542 +0000 UTC m=+37.532445583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : configmap references non-existent config key: service-ca.crt Sep 10 15:27:47.249359 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:47.249328 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:27:47.249557 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:47.249378 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:47.749369159 +0000 UTC m=+37.532582205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : secret "router-metrics-certs-default" not found Sep 10 15:27:47.251527 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.251506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-default-certificate\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.251629 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.251611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-stats-auth\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.267037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.266969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn47p\" (UniqueName: \"kubernetes.io/projected/16ae4a17-d168-4499-8387-639457cdb1d4-kube-api-access-hn47p\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.515374 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.515338 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-59q64"] Sep 10 15:27:47.519498 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:27:47.519320 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1d53ff_0078_41fe_9d4e_66c0629657a1.slice/crio-fc64892afc385afb5d9a855e9afe3c3d65bf8747937aeaa5d64f8fd97a4a57ef WatchSource:0}: Error finding container fc64892afc385afb5d9a855e9afe3c3d65bf8747937aeaa5d64f8fd97a4a57ef: Status 404 returned error can't find the container with id fc64892afc385afb5d9a855e9afe3c3d65bf8747937aeaa5d64f8fd97a4a57ef Sep 10 15:27:47.753665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.753629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.754154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.753704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:47.754154 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:47.753808 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:48.75378858 +0000 UTC m=+38.537001636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : configmap references non-existent config key: service-ca.crt Sep 10 15:27:47.754154 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:47.753816 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:27:47.754154 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:47.753852 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:48.753841356 +0000 UTC m=+38.537054398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : secret "router-metrics-certs-default" not found Sep 10 15:27:47.971687 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.971658 2581 generic.go:356] "Generic (PLEG): container finished" podID="e14e464b-2049-45bb-b740-7be313711c2c" containerID="0114caf569251c95bf7b01178fd794a799b8b0b09829fbc4a4ef1ddcf23b8b33" exitCode=0 Sep 10 15:27:47.972132 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.972087 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerDied","Data":"0114caf569251c95bf7b01178fd794a799b8b0b09829fbc4a4ef1ddcf23b8b33"} Sep 10 15:27:47.977132 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:47.977106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" event={"ID":"2e1d53ff-0078-41fe-9d4e-66c0629657a1","Type":"ContainerStarted","Data":"fc64892afc385afb5d9a855e9afe3c3d65bf8747937aeaa5d64f8fd97a4a57ef"} Sep 10 15:27:48.762321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:48.762218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:48.762360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:48.762386 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:48.762418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:48.762456 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:50.762436689 +0000 UTC m=+40.545649731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : secret "router-metrics-certs-default" not found Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:48.762468 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:48.762521 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls podName:956f0ede-4db9-4494-ad43-b99970036e50 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:52.762503006 +0000 UTC m=+42.545716065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls") pod "dns-default-wh8rn" (UID: "956f0ede-4db9-4494-ad43-b99970036e50") : secret "dns-default-metrics-tls" not found Sep 10 15:27:48.762723 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:48.762539 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:50.762527807 +0000 UTC m=+40.545740848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : configmap references non-existent config key: service-ca.crt Sep 10 15:27:48.983977 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:48.983945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" event={"ID":"e14e464b-2049-45bb-b740-7be313711c2c","Type":"ContainerStarted","Data":"f132c024bde56ef91e5721eab5894a84639e011f864ec36af0d0e92aac1e516a"} Sep 10 15:27:49.013939 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:49.013847 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wd2xg" podStartSLOduration=9.714755061 podStartE2EDuration="39.013832035s" totalImagesPullingTime="29.299076974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:49.011557098 +0000 UTC m=+38.794770175" watchObservedRunningTime="2026-09-10 15:27:49.013832035 +0000 UTC m=+38.797045099" Sep 10 15:27:50.780258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:50.780168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:50.780258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:50.780243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:50.780722 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:50.780328 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:27:50.780722 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:50.780376 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:54.780354302 +0000 UTC m=+44.563567357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : configmap references non-existent config key: service-ca.crt Sep 10 15:27:50.780722 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:50.780404 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:54.78039443 +0000 UTC m=+44.563607477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : secret "router-metrics-certs-default" not found Sep 10 15:27:50.989699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:50.989663 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zhxq" event={"ID":"3a1ae1ee-a086-4096-86d3-4ec6ef4672cd","Type":"ContainerStarted","Data":"6b6300aaf0fff82739b226dac172aabc63b41278032d24742c3df55728a77a65"} Sep 10 15:27:50.991176 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:50.991152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" event={"ID":"2e1d53ff-0078-41fe-9d4e-66c0629657a1","Type":"ContainerStarted","Data":"6ffa94847a01d24065d2f2a3e71a0ff86d28ff631c25632ca6dafc3505c277f1"} Sep 10 15:27:51.010029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.009971 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zhxq" podStartSLOduration=2.598943926 podStartE2EDuration="6.009958099s" totalImagesPullingTime="3.411014173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:51.009031739 +0000 UTC m=+40.792244805" watchObservedRunningTime="2026-09-10 15:27:51.009958099 +0000 UTC m=+40.793171162" Sep 10 15:27:51.026868 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.026817 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" podStartSLOduration=2.336361536 podStartE2EDuration="5.026797762s" totalImagesPullingTime="2.690436226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:51.025532491 +0000 UTC m=+40.808745556" watchObservedRunningTime="2026-09-10 15:27:51.026797762 +0000 UTC m=+40.810010831" Sep 10 15:27:51.467211 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.467180 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qmv8m"] Sep 10 15:27:51.498935 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.498905 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qmv8m"] Sep 10 15:27:51.499116 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.499061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.501232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.501211 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 15:27:51.501232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.501219 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 15:27:51.501409 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.501336 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrkll\"" Sep 10 15:27:51.587150 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.587122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zzv\" (UniqueName: \"kubernetes.io/projected/6911314a-52b9-47c0-be21-d844126dc91b-kube-api-access-q7zzv\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.587278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.587233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6911314a-52b9-47c0-be21-d844126dc91b-data-volume\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.587278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.587262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6911314a-52b9-47c0-be21-d844126dc91b-crio-socket\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.587344 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.587288 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.587344 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.587326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688434 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6911314a-52b9-47c0-be21-d844126dc91b-crio-socket\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688581 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688581 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688581 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zzv\" (UniqueName: \"kubernetes.io/projected/6911314a-52b9-47c0-be21-d844126dc91b-kube-api-access-q7zzv\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688581 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:51.688534 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:27:51.688716 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:51.688595 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:52.18857833 +0000 UTC m=+41.971791390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : configmap "kube-rbac-proxy" not found Sep 10 15:27:51.688716 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688601 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6911314a-52b9-47c0-be21-d844126dc91b-crio-socket\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688716 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6911314a-52b9-47c0-be21-d844126dc91b-data-volume\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.688716 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:51.688654 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:27:51.688716 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:51.688710 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:52.188695231 +0000 UTC m=+41.971908292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : secret "insights-runtime-extractor-tls" not found Sep 10 15:27:51.688913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.688884 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6911314a-52b9-47c0-be21-d844126dc91b-data-volume\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:51.698428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:51.698399 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zzv\" (UniqueName: \"kubernetes.io/projected/6911314a-52b9-47c0-be21-d844126dc91b-kube-api-access-q7zzv\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:52.066592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.066558 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59594956fd-7r98d"] Sep 10 15:27:52.086943 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.086917 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.091043 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.091018 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 15:27:52.091043 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.091032 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 15:27:52.091223 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.091055 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 15:27:52.092199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.092182 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f2kws\"" Sep 10 15:27:52.092860 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.092837 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59594956fd-7r98d"] Sep 10 15:27:52.098908 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.098888 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 15:27:52.192327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-certificates\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-installation-pull-secrets\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-image-registry-private-configuration\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-ca-trust-extracted\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mqp\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-kube-api-access-f6mqp\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192565 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-trusted-ca\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-bound-sa-token\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.192614 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.192637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.192662 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:53.192649112 +0000 UTC m=+42.975862154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : configmap "kube-rbac-proxy" not found Sep 10 15:27:52.192743 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.192744 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:27:52.192960 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.192804 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:53.192788987 +0000 UTC m=+42.976002043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : secret "insights-runtime-extractor-tls" not found Sep 10 15:27:52.293864 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.293828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-bound-sa-token\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.293884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-certificates\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.293909 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.293925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-installation-pull-secrets\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.293946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-image-registry-private-configuration\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.294009 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-ca-trust-extracted\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.294010 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:27:52.294074 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.294031 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59594956fd-7r98d: secret "image-registry-tls" not found Sep 10 15:27:52.294432 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.294092 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls podName:1ba091bb-0667-494e-b52e-0d7cb8bfd92b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:52.794075679 +0000 UTC m=+42.577288740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls") pod "image-registry-59594956fd-7r98d" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b") : secret "image-registry-tls" not found Sep 10 15:27:52.294432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.294233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mqp\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-kube-api-access-f6mqp\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.294262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-trusted-ca\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294598 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.294453 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-ca-trust-extracted\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294598 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.294553 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-certificates\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.294975 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.294943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-trusted-ca\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.296487 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.296467 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-image-registry-private-configuration\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.296545 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.296472 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-installation-pull-secrets\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.311840 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.311814 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mqp\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-kube-api-access-f6mqp\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.312183 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.312166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-bound-sa-token\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.798771 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.798734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:52.798983 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:52.798791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:27:52.798983 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.798883 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:27:52.798983 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.798905 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59594956fd-7r98d: secret "image-registry-tls" not found Sep 10 15:27:52.798983 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.798889 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:27:52.798983 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.798959 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls podName:956f0ede-4db9-4494-ad43-b99970036e50 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:00.798945572 +0000 UTC m=+50.582158613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls") pod "dns-default-wh8rn" (UID: "956f0ede-4db9-4494-ad43-b99970036e50") : secret "dns-default-metrics-tls" not found Sep 10 15:27:52.798983 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:52.798972 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls podName:1ba091bb-0667-494e-b52e-0d7cb8bfd92b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:53.798966856 +0000 UTC m=+43.582179898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls") pod "image-registry-59594956fd-7r98d" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b") : secret "image-registry-tls" not found Sep 10 15:27:53.202054 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:53.201969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:53.202448 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.202131 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:27:53.202448 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.202193 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:55.202178225 +0000 UTC m=+44.985391267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : secret "insights-runtime-extractor-tls" not found Sep 10 15:27:53.202448 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:53.202237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:53.202448 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.202309 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:27:53.202448 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.202333 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:55.202325506 +0000 UTC m=+44.985538547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : configmap "kube-rbac-proxy" not found Sep 10 15:27:53.396482 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:53.396455 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgfdg_1ea8ed65-1c50-40b1-8a47-ade9dd837594/dns-node-resolver/0.log" Sep 10 15:27:53.807501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:53.807464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:53.807657 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.807582 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:27:53.807657 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.807593 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59594956fd-7r98d: secret "image-registry-tls" not found Sep 10 15:27:53.807657 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:53.807641 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls podName:1ba091bb-0667-494e-b52e-0d7cb8bfd92b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:55.807628803 +0000 UTC m=+45.590841858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls") pod "image-registry-59594956fd-7r98d" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b") : secret "image-registry-tls" not found Sep 10 15:27:54.396445 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:54.396417 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5ph8_6cbb049d-4320-4bb3-8f6d-0deda397fd56/node-ca/0.log" Sep 10 15:27:54.815098 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:54.815066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:54.815258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:54.815124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:27:54.815258 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:54.815226 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:02.815208325 +0000 UTC m=+52.598421372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : configmap references non-existent config key: service-ca.crt Sep 10 15:27:54.815258 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:54.815238 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:27:54.815363 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:54.815284 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:02.815267855 +0000 UTC m=+52.598480914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : secret "router-metrics-certs-default" not found Sep 10 15:27:55.005506 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:55.005477 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zhxq_3a1ae1ee-a086-4096-86d3-4ec6ef4672cd/serve-healthcheck-canary/0.log" Sep 10 15:27:55.218301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:55.218217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:55.218301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:55.218274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:55.218529 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.218367 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:27:55.218529 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.218377 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:27:55.218529 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.218447 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:59.218426849 +0000 UTC m=+49.001639905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : configmap "kube-rbac-proxy" not found Sep 10 15:27:55.218529 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.218473 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:59.218456976 +0000 UTC m=+49.001670044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : secret "insights-runtime-extractor-tls" not found Sep 10 15:27:55.823181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:55.823141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:55.823585 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.823301 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:27:55.823585 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.823321 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59594956fd-7r98d: secret "image-registry-tls" not found Sep 10 15:27:55.823585 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:55.823375 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls podName:1ba091bb-0667-494e-b52e-0d7cb8bfd92b nodeName:}" failed. No retries permitted until 2026-09-10 15:27:59.82336039 +0000 UTC m=+49.606573432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls") pod "image-registry-59594956fd-7r98d" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b") : secret "image-registry-tls" not found Sep 10 15:27:59.252437 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:59.252396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:59.252831 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:59.252453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:27:59.252831 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.252520 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:27:59.252831 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.252538 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:27:59.252831 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.252587 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:28:07.252571415 +0000 UTC m=+57.035784458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : configmap "kube-rbac-proxy" not found Sep 10 15:27:59.252831 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.252656 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:28:07.252637434 +0000 UTC m=+57.035850481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : secret "insights-runtime-extractor-tls" not found Sep 10 15:27:59.859073 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:27:59.859030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:27:59.859279 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.859200 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:27:59.859279 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.859226 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59594956fd-7r98d: secret "image-registry-tls" not found Sep 10 15:27:59.859387 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:27:59.859294 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls podName:1ba091bb-0667-494e-b52e-0d7cb8bfd92b nodeName:}" failed. No retries permitted until 2026-09-10 15:28:07.85927336 +0000 UTC m=+57.642486409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls") pod "image-registry-59594956fd-7r98d" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b") : secret "image-registry-tls" not found Sep 10 15:28:00.866618 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:00.866580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:28:00.867093 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:00.866713 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:28:00.867093 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:00.866772 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls podName:956f0ede-4db9-4494-ad43-b99970036e50 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:16.866756099 +0000 UTC m=+66.649969141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls") pod "dns-default-wh8rn" (UID: "956f0ede-4db9-4494-ad43-b99970036e50") : secret "dns-default-metrics-tls" not found Sep 10 15:28:02.880120 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:02.880063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:02.880535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:02.880148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:02.880535 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:02.880243 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:28:02.880535 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:02.880249 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:18.880230306 +0000 UTC m=+68.663443349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : configmap references non-existent config key: service-ca.crt Sep 10 15:28:02.880535 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:02.880291 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs podName:16ae4a17-d168-4499-8387-639457cdb1d4 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:18.88027241 +0000 UTC m=+68.663485452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs") pod "router-default-7fd6757d9d-prmxg" (UID: "16ae4a17-d168-4499-8387-639457cdb1d4") : secret "router-metrics-certs-default" not found Sep 10 15:28:07.315247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:07.315190 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:28:07.315247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:07.315257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:28:07.315739 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:07.315369 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:28:07.315739 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:07.315448 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls podName:6911314a-52b9-47c0-be21-d844126dc91b nodeName:}" failed. No retries permitted until 2026-09-10 15:28:23.315428057 +0000 UTC m=+73.098641115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmv8m" (UID: "6911314a-52b9-47c0-be21-d844126dc91b") : secret "insights-runtime-extractor-tls" not found Sep 10 15:28:07.315739 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:07.315694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6911314a-52b9-47c0-be21-d844126dc91b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:28:07.919867 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:07.919825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:28:07.920081 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:07.919980 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:28:07.920081 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:07.920020 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59594956fd-7r98d: secret "image-registry-tls" not found Sep 10 15:28:07.920081 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:07.920080 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls podName:1ba091bb-0667-494e-b52e-0d7cb8bfd92b nodeName:}" failed. No retries permitted until 2026-09-10 15:28:23.920060745 +0000 UTC m=+73.703273804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls") pod "image-registry-59594956fd-7r98d" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b") : secret "image-registry-tls" not found Sep 10 15:28:10.955446 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:10.955420 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkm4r" Sep 10 15:28:16.486148 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.486096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:28:16.486148 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.486156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:28:16.486697 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.486197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:28:16.488716 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.488691 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:28:16.488827 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.488731 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:28:16.488827 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.488698 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:28:16.496754 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:16.496736 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 15:28:16.496847 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:16.496804 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs podName:e9fb2ac2-2577-4d59-ad17-cb67bd293b36 nodeName:}" failed. No retries permitted until 2026-09-10 15:29:20.496787813 +0000 UTC m=+130.280000854 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs") pod "network-metrics-daemon-9rlrq" (UID: "e9fb2ac2-2577-4d59-ad17-cb67bd293b36") : secret "metrics-daemon-secret" not found Sep 10 15:28:16.498552 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.498535 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:28:16.498687 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.498671 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1-original-pull-secret\") pod \"global-pull-secret-syncer-4wc8d\" (UID: \"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1\") " pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:28:16.504341 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.504327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wc8d" Sep 10 15:28:16.509509 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.509486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qc8l\" (UniqueName: \"kubernetes.io/projected/2b83d159-086f-4d5a-ae13-ba013a2490af-kube-api-access-7qc8l\") pod \"network-check-target-q5rkh\" (UID: \"2b83d159-086f-4d5a-ae13-ba013a2490af\") " pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:28:16.512440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.512414 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rkds\"" Sep 10 15:28:16.521192 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.521177 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:28:16.805701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.805669 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4wc8d"] Sep 10 15:28:16.809826 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:16.809799 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5fd8d8_fe5b_4123_ae3e_c5a780cab6e1.slice/crio-a351cd7210b13d6c30858b112708f73e143d7cde5d1c9cc37866f430c1943fea WatchSource:0}: Error finding container a351cd7210b13d6c30858b112708f73e143d7cde5d1c9cc37866f430c1943fea: Status 404 returned error can't find the container with id a351cd7210b13d6c30858b112708f73e143d7cde5d1c9cc37866f430c1943fea Sep 10 15:28:16.832012 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.831968 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5rkh"] Sep 10 15:28:16.834858 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:16.834837 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b83d159_086f_4d5a_ae13_ba013a2490af.slice/crio-c5e1824d76d87d22662184ef14001634d402b6ace7089491dfb1783eb9561fd4 WatchSource:0}: Error finding container c5e1824d76d87d22662184ef14001634d402b6ace7089491dfb1783eb9561fd4: Status 404 returned error can't find the container with id c5e1824d76d87d22662184ef14001634d402b6ace7089491dfb1783eb9561fd4 Sep 10 15:28:16.890617 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.890586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:28:16.892816 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:16.892796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/956f0ede-4db9-4494-ad43-b99970036e50-metrics-tls\") pod \"dns-default-wh8rn\" (UID: \"956f0ede-4db9-4494-ad43-b99970036e50\") " pod="openshift-dns/dns-default-wh8rn" Sep 10 15:28:17.054973 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:17.054941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4wc8d" event={"ID":"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1","Type":"ContainerStarted","Data":"a351cd7210b13d6c30858b112708f73e143d7cde5d1c9cc37866f430c1943fea"} Sep 10 15:28:17.056124 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:17.056064 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5rkh" event={"ID":"2b83d159-086f-4d5a-ae13-ba013a2490af","Type":"ContainerStarted","Data":"c5e1824d76d87d22662184ef14001634d402b6ace7089491dfb1783eb9561fd4"} Sep 10 15:28:17.159218 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:17.159186 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w5mt4\"" Sep 10 15:28:17.167154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:17.167132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wh8rn" Sep 10 15:28:17.475346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:17.475276 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wh8rn"] Sep 10 15:28:17.480617 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:17.480584 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956f0ede_4db9_4494_ad43_b99970036e50.slice/crio-c4db57d83ca79d795ab678d1fd7028d6fdfdfc03d225f0619be6329fd785fd8c WatchSource:0}: Error finding container c4db57d83ca79d795ab678d1fd7028d6fdfdfc03d225f0619be6329fd785fd8c: Status 404 returned error can't find the container with id c4db57d83ca79d795ab678d1fd7028d6fdfdfc03d225f0619be6329fd785fd8c Sep 10 15:28:18.061565 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:18.061472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wh8rn" event={"ID":"956f0ede-4db9-4494-ad43-b99970036e50","Type":"ContainerStarted","Data":"c4db57d83ca79d795ab678d1fd7028d6fdfdfc03d225f0619be6329fd785fd8c"} Sep 10 15:28:18.909200 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:18.909156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:18.909386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:18.909275 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:18.910127 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:18.910096 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ae4a17-d168-4499-8387-639457cdb1d4-service-ca-bundle\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:18.912749 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:18.912724 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ae4a17-d168-4499-8387-639457cdb1d4-metrics-certs\") pod \"router-default-7fd6757d9d-prmxg\" (UID: \"16ae4a17-d168-4499-8387-639457cdb1d4\") " pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:19.121348 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:19.121318 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2bpf8\"" Sep 10 15:28:19.129584 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:19.129477 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:21.947809 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:21.947724 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fd6757d9d-prmxg"] Sep 10 15:28:21.952209 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:21.952183 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ae4a17_d168_4499_8387_639457cdb1d4.slice/crio-42590d801890bf07e1c41843ad1d7a661ca64a3f1e91bc122c248cc56ecc0ac4 WatchSource:0}: Error finding container 42590d801890bf07e1c41843ad1d7a661ca64a3f1e91bc122c248cc56ecc0ac4: Status 404 returned error can't find the container with id 42590d801890bf07e1c41843ad1d7a661ca64a3f1e91bc122c248cc56ecc0ac4 Sep 10 15:28:22.077461 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.077421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wh8rn" event={"ID":"956f0ede-4db9-4494-ad43-b99970036e50","Type":"ContainerStarted","Data":"3a37c25e2ca10d963cddffc17f105fbf4ae7df7858999d878c11d92f029848c8"} Sep 10 15:28:22.077676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.077478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wh8rn" event={"ID":"956f0ede-4db9-4494-ad43-b99970036e50","Type":"ContainerStarted","Data":"d955c6e0de5faed062cb67b0072f2d6f4a19f395d9df7cfc6285a1deb2acd990"} Sep 10 15:28:22.077676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.077514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wh8rn" Sep 10 15:28:22.079032 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.078987 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5rkh" event={"ID":"2b83d159-086f-4d5a-ae13-ba013a2490af","Type":"ContainerStarted","Data":"1c619d96845244916a5381a52bee3394cafcb0f00b99dd5748b81ac8a1b5f92b"} Sep 10 15:28:22.079156 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.079083 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:28:22.080338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.080320 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4wc8d" event={"ID":"2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1","Type":"ContainerStarted","Data":"970bd680a881d60a9c4048cb08f43f0bade767ff65cbeb6761662bee6821f12e"} Sep 10 15:28:22.081762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.081743 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" event={"ID":"16ae4a17-d168-4499-8387-639457cdb1d4","Type":"ContainerStarted","Data":"8bb106f86d8af44bf82f696ddd117c91767a59449d4d5efc46c6eeb7baff95f8"} Sep 10 15:28:22.081860 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.081777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" event={"ID":"16ae4a17-d168-4499-8387-639457cdb1d4","Type":"ContainerStarted","Data":"42590d801890bf07e1c41843ad1d7a661ca64a3f1e91bc122c248cc56ecc0ac4"} Sep 10 15:28:22.097917 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.097871 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wh8rn" podStartSLOduration=32.906330103 podStartE2EDuration="37.097856856s" totalImagesPullingTime="4.191526753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:22.097375138 +0000 UTC m=+71.880588202" watchObservedRunningTime="2026-09-10 15:28:22.097856856 +0000 UTC m=+71.881069919" Sep 10 15:28:22.117108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.117066 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4wc8d" podStartSLOduration=66.23936392 podStartE2EDuration="1m11.117051885s" totalImagesPullingTime="4.877687965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:22.116494327 +0000 UTC m=+71.899707391" watchObservedRunningTime="2026-09-10 15:28:22.117051885 +0000 UTC m=+71.900264951" Sep 10 15:28:22.130156 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.130131 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:22.135722 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.135696 2581 patch_prober.go:28] interesting pod/router-default-7fd6757d9d-prmxg container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Sep 10 15:28:22.135722 ip-10-0-141-82 kubenswrapper[2581]: [+]has-synced ok Sep 10 15:28:22.135722 ip-10-0-141-82 kubenswrapper[2581]: [+]process-running ok Sep 10 15:28:22.135722 ip-10-0-141-82 kubenswrapper[2581]: healthz check failed Sep 10 15:28:22.135982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.135746 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" podUID="16ae4a17-d168-4499-8387-639457cdb1d4" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 15:28:22.138338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.138301 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q5rkh" podStartSLOduration=67.303358141 podStartE2EDuration="1m12.138289586s" totalImagesPullingTime="4.834931445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:22.137247423 +0000 UTC m=+71.920460487" watchObservedRunningTime="2026-09-10 15:28:22.138289586 +0000 UTC m=+71.921502649" Sep 10 15:28:22.159321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:22.159270 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" podStartSLOduration=36.159249862 podStartE2EDuration="36.159249862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:22.158962958 +0000 UTC m=+71.942176022" watchObservedRunningTime="2026-09-10 15:28:22.159249862 +0000 UTC m=+71.942462927" Sep 10 15:28:23.105249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.105215 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59594956fd-7r98d"] Sep 10 15:28:23.105662 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:23.105443 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-59594956fd-7r98d" podUID="1ba091bb-0667-494e-b52e-0d7cb8bfd92b" Sep 10 15:28:23.130981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.130947 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:23.174678 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.174638 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-ch5hg"] Sep 10 15:28:23.207198 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.207166 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x"] Sep 10 15:28:23.207357 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.207280 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.210662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.210632 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5cz4b\"" Sep 10 15:28:23.210925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.210901 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 15:28:23.211049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.210938 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 15:28:23.225500 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.225476 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68cc9cc884-t9l6k"] Sep 10 15:28:23.225601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.225526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:23.230212 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.230188 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gq6sq\"" Sep 10 15:28:23.230212 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.230210 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 15:28:23.239382 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.239363 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x"] Sep 10 15:28:23.239464 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.239391 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-ch5hg"] Sep 10 15:28:23.239464 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.239407 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68cc9cc884-t9l6k"] Sep 10 15:28:23.239530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.239477 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.242386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.242359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3665a98-3024-4e1a-bf29-ac220d83c587-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-ch5hg\" (UID: \"a3665a98-3024-4e1a-bf29-ac220d83c587\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.242506 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.242489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a3665a98-3024-4e1a-bf29-ac220d83c587-nginx-conf\") pod \"networking-console-plugin-79977f9844-ch5hg\" (UID: \"a3665a98-3024-4e1a-bf29-ac220d83c587\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.342893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.342857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3665a98-3024-4e1a-bf29-ac220d83c587-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-ch5hg\" (UID: \"a3665a98-3024-4e1a-bf29-ac220d83c587\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.342893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.342896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-bound-sa-token\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.342914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-ca-trust-extracted\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.342932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-installation-pull-secrets\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.342991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:28:23.343165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-trusted-ca\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-image-registry-private-configuration\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a3665a98-3024-4e1a-bf29-ac220d83c587-nginx-conf\") pod \"networking-console-plugin-79977f9844-ch5hg\" (UID: \"a3665a98-3024-4e1a-bf29-ac220d83c587\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.343410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-registry-tls\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea1fe005-48d7-4140-9d69-4e6873fc7c49-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-nms9x\" (UID: \"ea1fe005-48d7-4140-9d69-4e6873fc7c49\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:23.343410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hvp\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-kube-api-access-c2hvp\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-registry-certificates\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.343840 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.343817 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a3665a98-3024-4e1a-bf29-ac220d83c587-nginx-conf\") pod \"networking-console-plugin-79977f9844-ch5hg\" (UID: \"a3665a98-3024-4e1a-bf29-ac220d83c587\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.345284 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.345259 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6911314a-52b9-47c0-be21-d844126dc91b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmv8m\" (UID: \"6911314a-52b9-47c0-be21-d844126dc91b\") " pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:28:23.345377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.345363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3665a98-3024-4e1a-bf29-ac220d83c587-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-ch5hg\" (UID: \"a3665a98-3024-4e1a-bf29-ac220d83c587\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.444727 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-bound-sa-token\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.444727 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-ca-trust-extracted\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.444727 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-installation-pull-secrets\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-trusted-ca\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-image-registry-private-configuration\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-registry-tls\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea1fe005-48d7-4140-9d69-4e6873fc7c49-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-nms9x\" (UID: \"ea1fe005-48d7-4140-9d69-4e6873fc7c49\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:23.445029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.444981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hvp\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-kube-api-access-c2hvp\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445190 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.445042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-registry-certificates\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445190 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.445126 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-ca-trust-extracted\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445704 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.445676 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-registry-certificates\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.445846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.445799 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-trusted-ca\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.447495 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.447474 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-installation-pull-secrets\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.447572 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.447557 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-image-registry-private-configuration\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.447775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.447756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ea1fe005-48d7-4140-9d69-4e6873fc7c49-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-nms9x\" (UID: \"ea1fe005-48d7-4140-9d69-4e6873fc7c49\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:23.447835 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.447771 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-registry-tls\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.455850 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.455818 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hvp\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-kube-api-access-c2hvp\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.456217 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.456201 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8-bound-sa-token\") pod \"image-registry-68cc9cc884-t9l6k\" (UID: \"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8\") " pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.515920 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.515890 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" Sep 10 15:28:23.533989 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.533968 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:23.549393 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.549367 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f2kws\"" Sep 10 15:28:23.558077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.558049 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:23.611028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.610988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrkll\"" Sep 10 15:28:23.619904 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.619641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qmv8m" Sep 10 15:28:23.806014 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.805968 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-ch5hg"] Sep 10 15:28:23.808801 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:23.808761 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3665a98_3024_4e1a_bf29_ac220d83c587.slice/crio-8d79d578cc6bb468351d4fa1981fd76f758358a8d7274169ad994dbb4dfe0b0b WatchSource:0}: Error finding container 8d79d578cc6bb468351d4fa1981fd76f758358a8d7274169ad994dbb4dfe0b0b: Status 404 returned error can't find the container with id 8d79d578cc6bb468351d4fa1981fd76f758358a8d7274169ad994dbb4dfe0b0b Sep 10 15:28:23.830800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.830775 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x"] Sep 10 15:28:23.833490 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:23.833452 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1fe005_48d7_4140_9d69_4e6873fc7c49.slice/crio-8d9a1136eb7c13dc008832169d6e5773ff5dc4290ab8929894ccbc242f910d31 WatchSource:0}: Error finding container 8d9a1136eb7c13dc008832169d6e5773ff5dc4290ab8929894ccbc242f910d31: Status 404 returned error can't find the container with id 8d9a1136eb7c13dc008832169d6e5773ff5dc4290ab8929894ccbc242f910d31 Sep 10 15:28:23.859114 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.859090 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68cc9cc884-t9l6k"] Sep 10 15:28:23.861881 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:23.861858 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c2445d_2f2a_48b9_b5c9_d4a5d9717aa8.slice/crio-5ff495cf188ea2ed9a307ebb032b49ac253740e4f4feab5fb089cb394378c567 WatchSource:0}: Error finding container 5ff495cf188ea2ed9a307ebb032b49ac253740e4f4feab5fb089cb394378c567: Status 404 returned error can't find the container with id 5ff495cf188ea2ed9a307ebb032b49ac253740e4f4feab5fb089cb394378c567 Sep 10 15:28:23.912682 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.912657 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qmv8m"] Sep 10 15:28:23.920959 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:23.920933 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6911314a_52b9_47c0_be21_d844126dc91b.slice/crio-641cd243e6009bb3b0047461f0a23688ac172dca8d3227f3c0a5b6e231f55936 WatchSource:0}: Error finding container 641cd243e6009bb3b0047461f0a23688ac172dca8d3227f3c0a5b6e231f55936: Status 404 returned error can't find the container with id 641cd243e6009bb3b0047461f0a23688ac172dca8d3227f3c0a5b6e231f55936 Sep 10 15:28:23.950021 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.949980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:28:23.952207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:23.952185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"image-registry-59594956fd-7r98d\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:28:24.091361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.091274 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" event={"ID":"ea1fe005-48d7-4140-9d69-4e6873fc7c49","Type":"ContainerStarted","Data":"8d9a1136eb7c13dc008832169d6e5773ff5dc4290ab8929894ccbc242f910d31"} Sep 10 15:28:24.092775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.092749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmv8m" event={"ID":"6911314a-52b9-47c0-be21-d844126dc91b","Type":"ContainerStarted","Data":"b3c25d2af7bd0e19528d720d8c6e58c6315bcd4ab745de31f3d773250bc0ba1b"} Sep 10 15:28:24.092925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.092795 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmv8m" event={"ID":"6911314a-52b9-47c0-be21-d844126dc91b","Type":"ContainerStarted","Data":"641cd243e6009bb3b0047461f0a23688ac172dca8d3227f3c0a5b6e231f55936"} Sep 10 15:28:24.093958 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.093934 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" event={"ID":"a3665a98-3024-4e1a-bf29-ac220d83c587","Type":"ContainerStarted","Data":"8d79d578cc6bb468351d4fa1981fd76f758358a8d7274169ad994dbb4dfe0b0b"} Sep 10 15:28:24.095348 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.095320 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" event={"ID":"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8","Type":"ContainerStarted","Data":"1574cfd9aa7fcff07b01f64beba02f3dde824f9f8fcfcc0dcb384ec14a453cbf"} Sep 10 15:28:24.095441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.095359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:28:24.095441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.095368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" event={"ID":"e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8","Type":"ContainerStarted","Data":"5ff495cf188ea2ed9a307ebb032b49ac253740e4f4feab5fb089cb394378c567"} Sep 10 15:28:24.095636 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.095618 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:24.096848 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.096833 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7fd6757d9d-prmxg" Sep 10 15:28:24.101776 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.101761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:28:24.151587 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151555 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-installation-pull-secrets\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151601 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-ca-trust-extracted\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151651 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-bound-sa-token\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151683 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-image-registry-private-configuration\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151709 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6mqp\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-kube-api-access-f6mqp\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151858 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-certificates\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151898 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.151937 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-trusted-ca\") pod \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\" (UID: \"1ba091bb-0667-494e-b52e-0d7cb8bfd92b\") " Sep 10 15:28:24.152419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.152305 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:28:24.152476 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.152425 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:28:24.153475 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.152852 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:28:24.153475 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.152956 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-certificates\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.153475 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.152975 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-trusted-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.153475 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.153009 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-ca-trust-extracted\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.155634 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.155609 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:28:24.155741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.155675 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:28:24.155741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.155730 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:28:24.155741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.155726 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-kube-api-access-f6mqp" (OuterVolumeSpecName: "kube-api-access-f6mqp") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "kube-api-access-f6mqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:28:24.155910 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.155855 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1ba091bb-0667-494e-b52e-0d7cb8bfd92b" (UID: "1ba091bb-0667-494e-b52e-0d7cb8bfd92b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:28:24.158517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.158465 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" podStartSLOduration=1.158449754 podStartE2EDuration="1.158449754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:24.156943618 +0000 UTC m=+73.940156684" watchObservedRunningTime="2026-09-10 15:28:24.158449754 +0000 UTC m=+73.941662817" Sep 10 15:28:24.253699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.253668 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-installation-pull-secrets\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.253699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.253696 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-bound-sa-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.253699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.253705 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-image-registry-private-configuration\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.253948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.253714 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6mqp\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-kube-api-access-f6mqp\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:24.253948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:24.253722 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ba091bb-0667-494e-b52e-0d7cb8bfd92b-registry-tls\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:28:25.099085 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:25.099035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59594956fd-7r98d" Sep 10 15:28:25.099388 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:25.099365 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:25.134016 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:25.133974 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59594956fd-7r98d"] Sep 10 15:28:25.136107 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:25.136078 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59594956fd-7r98d"] Sep 10 15:28:26.104095 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:26.104057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmv8m" event={"ID":"6911314a-52b9-47c0-be21-d844126dc91b","Type":"ContainerStarted","Data":"93c3b5acfe501e6776ece630db03266fb7d22f222aa15f26b44a911c5ebaece5"} Sep 10 15:28:26.105493 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:26.105469 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" event={"ID":"a3665a98-3024-4e1a-bf29-ac220d83c587","Type":"ContainerStarted","Data":"09933496571b44a61dd6e312f0346b88f243a4f2ffe1b6e6d32f87a1aaa57b2d"} Sep 10 15:28:26.106873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:26.106848 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" event={"ID":"ea1fe005-48d7-4140-9d69-4e6873fc7c49","Type":"ContainerStarted","Data":"25d2c26a958542c972f5d1af27936feb4758d20a1e4d3298d2bb0b6a954ad411"} Sep 10 15:28:26.121898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:26.121853 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-ch5hg" podStartSLOduration=1.6842785120000001 podStartE2EDuration="3.121840731s" totalImagesPullingTime="1.437562219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:26.120687047 +0000 UTC m=+75.903900113" watchObservedRunningTime="2026-09-10 15:28:26.121840731 +0000 UTC m=+75.905053822" Sep 10 15:28:26.786223 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:26.786185 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba091bb-0667-494e-b52e-0d7cb8bfd92b" path="/var/lib/kubelet/pods/1ba091bb-0667-494e-b52e-0d7cb8bfd92b/volumes" Sep 10 15:28:27.112450 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.112413 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmv8m" event={"ID":"6911314a-52b9-47c0-be21-d844126dc91b","Type":"ContainerStarted","Data":"35a96a777abe40217732eb10f2b6b6de2eccfec825b4dfc142e40c4797926802"} Sep 10 15:28:27.112813 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.112725 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:27.117104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.117083 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" Sep 10 15:28:27.133304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.133262 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qmv8m" podStartSLOduration=33.264122934 podStartE2EDuration="36.133249968s" totalImagesPullingTime="2.869127034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:27.131894298 +0000 UTC m=+76.915107362" watchObservedRunningTime="2026-09-10 15:28:27.133249968 +0000 UTC m=+76.916463031" Sep 10 15:28:27.133471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.133453 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-nms9x" podStartSLOduration=2.717809139 podStartE2EDuration="4.133449553s" totalImagesPullingTime="1.415640414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:26.136806724 +0000 UTC m=+75.920019788" watchObservedRunningTime="2026-09-10 15:28:27.133449553 +0000 UTC m=+76.916662617" Sep 10 15:28:27.424324 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.424237 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-njh9m"] Sep 10 15:28:27.426849 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.426833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.429205 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.429180 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 15:28:27.429349 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.429327 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 15:28:27.429629 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.429614 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 15:28:27.430243 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.430228 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nvzqr\"" Sep 10 15:28:27.430243 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.430238 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 15:28:27.430368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.430248 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 15:28:27.442114 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.442092 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-njh9m"] Sep 10 15:28:27.481541 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.481507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.481682 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.481548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.481682 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.481579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btq92\" (UniqueName: \"kubernetes.io/projected/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-kube-api-access-btq92\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.481682 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.481623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.582156 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.582121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.582329 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.582168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.582329 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.582198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btq92\" (UniqueName: \"kubernetes.io/projected/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-kube-api-access-btq92\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.582329 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.582219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.582474 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:27.582376 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 10 15:28:27.582474 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:27.582447 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-tls podName:71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:28.082427602 +0000 UTC m=+77.865640658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-njh9m" (UID: "71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9") : secret "prometheus-operator-tls" not found Sep 10 15:28:27.582865 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.582845 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.584477 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.584453 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:27.593776 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:27.593756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btq92\" (UniqueName: \"kubernetes.io/projected/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-kube-api-access-btq92\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:28.087135 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:28.087090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:28.089470 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:28.089442 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-njh9m\" (UID: \"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:28.335353 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:28.335304 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" Sep 10 15:28:28.603782 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:28.603712 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-njh9m"] Sep 10 15:28:28.606805 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:28.606780 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d60ed7_85e3_4f8a_98e0_2a5aaf5707b9.slice/crio-45a98410c4acfe4e0270a2bc1f6ca82fbb4ac6334a1b41547c5f23087e8fc4ed WatchSource:0}: Error finding container 45a98410c4acfe4e0270a2bc1f6ca82fbb4ac6334a1b41547c5f23087e8fc4ed: Status 404 returned error can't find the container with id 45a98410c4acfe4e0270a2bc1f6ca82fbb4ac6334a1b41547c5f23087e8fc4ed Sep 10 15:28:29.121504 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:29.121465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" event={"ID":"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9","Type":"ContainerStarted","Data":"45a98410c4acfe4e0270a2bc1f6ca82fbb4ac6334a1b41547c5f23087e8fc4ed"} Sep 10 15:28:30.127705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:30.127672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" event={"ID":"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9","Type":"ContainerStarted","Data":"253c3917b5d71b8d1dc771509201fa9e923f5d4194e158a1815bc5d16e74ab28"} Sep 10 15:28:31.132570 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:31.132533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" event={"ID":"71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9","Type":"ContainerStarted","Data":"33f3662e8a61900fc8f29bf87f1d85d988d820df147f063f330ad14322588abe"} Sep 10 15:28:31.154114 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:31.154027 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-njh9m" podStartSLOduration=2.709890893 podStartE2EDuration="4.154008773s" totalImagesPullingTime="1.44411788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:31.153358495 +0000 UTC m=+80.936571561" watchObservedRunningTime="2026-09-10 15:28:31.154008773 +0000 UTC m=+80.937222029" Sep 10 15:28:32.088599 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.088565 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wh8rn" Sep 10 15:28:32.843779 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.843747 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m"] Sep 10 15:28:32.847646 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.847630 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:32.849892 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.849862 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 15:28:32.850030 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.849892 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 15:28:32.850030 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.849899 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dggcm\"" Sep 10 15:28:32.860537 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.860517 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m"] Sep 10 15:28:32.886247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.886217 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bk725"] Sep 10 15:28:32.889130 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.889110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:32.891122 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.891105 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 15:28:32.891739 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.891718 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 15:28:32.891819 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.891745 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nl9w4\"" Sep 10 15:28:32.891819 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.891745 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 15:28:32.924943 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.924919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:32.925047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.924948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62fd9543-387c-4a0e-be77-3662e87014e0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:32.925093 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.925046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwtmv\" (UniqueName: \"kubernetes.io/projected/62fd9543-387c-4a0e-be77-3662e87014e0-kube-api-access-dwtmv\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:32.925093 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:32.925066 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.026188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddls\" (UniqueName: \"kubernetes.io/projected/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-kube-api-access-lddls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-textfile\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026139 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-metrics-client-ca\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-sys\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026231 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwtmv\" (UniqueName: \"kubernetes.io/projected/62fd9543-387c-4a0e-be77-3662e87014e0-kube-api-access-dwtmv\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.026405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026308 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-tls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.026405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026379 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-root\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.026672 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:33.026414 2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 15:28:33.026672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62fd9543-387c-4a0e-be77-3662e87014e0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.026672 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:33.026507 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-tls podName:62fd9543-387c-4a0e-be77-3662e87014e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:33.52648864 +0000 UTC m=+83.309701685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-dnt8m" (UID: "62fd9543-387c-4a0e-be77-3662e87014e0") : secret "openshift-state-metrics-tls" not found Sep 10 15:28:33.026672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-wtmp\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.026672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.026564 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.027211 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.027184 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62fd9543-387c-4a0e-be77-3662e87014e0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.028719 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.028697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.037700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.037676 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwtmv\" (UniqueName: \"kubernetes.io/projected/62fd9543-387c-4a0e-be77-3662e87014e0-kube-api-access-dwtmv\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.127506 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127506 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-tls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127683 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:33.127562 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 15:28:33.127683 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:33.127622 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-tls podName:38c0535a-7a83-4ab9-b450-cb9e4024ab7d nodeName:}" failed. No retries permitted until 2026-09-10 15:28:33.627607842 +0000 UTC m=+83.410820884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-tls") pod "node-exporter-bk725" (UID: "38c0535a-7a83-4ab9-b450-cb9e4024ab7d") : secret "node-exporter-tls" not found Sep 10 15:28:33.127683 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-root\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-wtmp\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127731 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-root\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddls\" (UniqueName: \"kubernetes.io/projected/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-kube-api-access-lddls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.127832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127815 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-wtmp\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-textfile\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-metrics-client-ca\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-sys\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.127980 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-sys\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.128174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-textfile\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128397 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.128376 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-metrics-client-ca\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.128434 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.128376 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.129839 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.129822 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.139224 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.139198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddls\" (UniqueName: \"kubernetes.io/projected/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-kube-api-access-lddls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.531826 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.531741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.534188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.534166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fd9543-387c-4a0e-be77-3662e87014e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dnt8m\" (UID: \"62fd9543-387c-4a0e-be77-3662e87014e0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.632227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.632193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-tls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.634368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.634349 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c0535a-7a83-4ab9-b450-cb9e4024ab7d-node-exporter-tls\") pod \"node-exporter-bk725\" (UID: \"38c0535a-7a83-4ab9-b450-cb9e4024ab7d\") " pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.755958 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.755932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" Sep 10 15:28:33.797617 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.797591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bk725" Sep 10 15:28:33.809131 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:33.809096 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c0535a_7a83_4ab9_b450_cb9e4024ab7d.slice/crio-c45d9a1d65151abb77d617bf6e68cf0e1deef74364debd9bd0bb7c285a8356d8 WatchSource:0}: Error finding container c45d9a1d65151abb77d617bf6e68cf0e1deef74364debd9bd0bb7c285a8356d8: Status 404 returned error can't find the container with id c45d9a1d65151abb77d617bf6e68cf0e1deef74364debd9bd0bb7c285a8356d8 Sep 10 15:28:33.902896 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.902866 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:28:33.907306 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.907283 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:33.909715 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.909676 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 15:28:33.909715 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.909708 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hprwv\"" Sep 10 15:28:33.909893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.909748 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 15:28:33.909893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.909807 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 15:28:33.909893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.909837 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 15:28:33.910353 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.910332 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 15:28:33.910494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.910383 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 15:28:33.910494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.910406 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 15:28:33.910494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.910414 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 15:28:33.910494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.910386 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 15:28:33.919694 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:33.919674 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:28:34.029812 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.029780 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m"] Sep 10 15:28:34.032863 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:34.032839 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fd9543_387c_4a0e_be77_3662e87014e0.slice/crio-aa6bcd90809c92e94ea0ebb3b63c401b50400a6e8466ba9213fc41531f1ac0b4 WatchSource:0}: Error finding container aa6bcd90809c92e94ea0ebb3b63c401b50400a6e8466ba9213fc41531f1ac0b4: Status 404 returned error can't find the container with id aa6bcd90809c92e94ea0ebb3b63c401b50400a6e8466ba9213fc41531f1ac0b4 Sep 10 15:28:34.035101 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbgk\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-kube-api-access-wrbgk\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035162 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035520 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035580 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-web-config\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035580 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.035647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.035620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-out\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136120 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-out\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136245 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbgk\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-kube-api-access-wrbgk\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136245 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136245 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136245 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136249 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136454 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:34.136382 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 10 15:28:34.136454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136762 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:34.136467 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls podName:fb689dca-0e29-4f7e-abd1-317b55e6a908 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:34.636446888 +0000 UTC m=+84.419659946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908") : secret "alertmanager-main-tls" not found Sep 10 15:28:34.136762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-web-config\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.136762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.136583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.137653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.137503 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.137653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.137506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.137784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.137749 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.139223 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.139198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.139655 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.139627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.140413 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.140338 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.140413 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.140335 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.141106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.141081 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-out\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.141210 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.141176 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-web-config\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.141347 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.141326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.141410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.141331 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.144408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.144386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bk725" event={"ID":"38c0535a-7a83-4ab9-b450-cb9e4024ab7d","Type":"ContainerStarted","Data":"c45d9a1d65151abb77d617bf6e68cf0e1deef74364debd9bd0bb7c285a8356d8"} Sep 10 15:28:34.145247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.145230 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbgk\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-kube-api-access-wrbgk\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.146279 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.146259 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" event={"ID":"62fd9543-387c-4a0e-be77-3662e87014e0","Type":"ContainerStarted","Data":"93e065b60cce9678a4656859ab1206f99d46a7b932a3fb88dca8d944055be8a8"} Sep 10 15:28:34.146373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.146307 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" event={"ID":"62fd9543-387c-4a0e-be77-3662e87014e0","Type":"ContainerStarted","Data":"aa6bcd90809c92e94ea0ebb3b63c401b50400a6e8466ba9213fc41531f1ac0b4"} Sep 10 15:28:34.642096 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.642059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.644883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.644853 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.816832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.816798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:28:34.927203 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.927118 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw"] Sep 10 15:28:34.932099 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.932078 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:34.934641 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.934619 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g2jdq\"" Sep 10 15:28:34.934762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.934700 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 15:28:34.934762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.934673 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-77eb9694d2jv\"" Sep 10 15:28:34.934863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.934686 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 15:28:34.934863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.934702 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 15:28:34.934863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.934645 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 15:28:34.935420 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.935401 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 15:28:34.946300 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:34.946278 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw"] Sep 10 15:28:35.045432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-tls\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8ch\" (UniqueName: \"kubernetes.io/projected/ec489482-f04f-4991-9443-4b64fda06fe2-kube-api-access-9h8ch\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-grpc-tls\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.045841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.045730 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec489482-f04f-4991-9443-4b64fda06fe2-metrics-client-ca\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147013 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.146951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8ch\" (UniqueName: \"kubernetes.io/projected/ec489482-f04f-4991-9443-4b64fda06fe2-kube-api-access-9h8ch\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-grpc-tls\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec489482-f04f-4991-9443-4b64fda06fe2-metrics-client-ca\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.147415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.147240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-tls\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.148456 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.148382 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec489482-f04f-4991-9443-4b64fda06fe2-metrics-client-ca\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.150199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.150165 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.150603 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.150578 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.151134 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.151084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.151290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.151266 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.151403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.151380 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-thanos-querier-tls\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.152105 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.152079 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec489482-f04f-4991-9443-4b64fda06fe2-secret-grpc-tls\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.153518 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.153498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" event={"ID":"62fd9543-387c-4a0e-be77-3662e87014e0","Type":"ContainerStarted","Data":"b83cb21f164c90fb3f1c2cb178b7d0d6279df1717959d79075bb8c5c60628e70"} Sep 10 15:28:35.156209 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.156189 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8ch\" (UniqueName: \"kubernetes.io/projected/ec489482-f04f-4991-9443-4b64fda06fe2-kube-api-access-9h8ch\") pod \"thanos-querier-9d5bdb4b6-gj4tw\" (UID: \"ec489482-f04f-4991-9443-4b64fda06fe2\") " pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.242671 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.242639 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:35.518204 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.518088 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:28:35.521242 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:35.521218 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb689dca_0e29_4f7e_abd1_317b55e6a908.slice/crio-6bc7b270fa8dcaa21ce705fbe3d5251737da0007730e96269d8887621103e334 WatchSource:0}: Error finding container 6bc7b270fa8dcaa21ce705fbe3d5251737da0007730e96269d8887621103e334: Status 404 returned error can't find the container with id 6bc7b270fa8dcaa21ce705fbe3d5251737da0007730e96269d8887621103e334 Sep 10 15:28:35.527441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:35.527418 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw"] Sep 10 15:28:35.531609 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:35.531589 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec489482_f04f_4991_9443_4b64fda06fe2.slice/crio-a724ae05e895dbd70102d6afd37895be533f74f51597a9cd4ed39f2f96d0ea91 WatchSource:0}: Error finding container a724ae05e895dbd70102d6afd37895be533f74f51597a9cd4ed39f2f96d0ea91: Status 404 returned error can't find the container with id a724ae05e895dbd70102d6afd37895be533f74f51597a9cd4ed39f2f96d0ea91 Sep 10 15:28:36.160208 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:36.160171 2581 generic.go:356] "Generic (PLEG): container finished" podID="38c0535a-7a83-4ab9-b450-cb9e4024ab7d" containerID="6a555fbafa9683d2b12ddcef02ac9b491ba7ae3c4a9dac334cae836d724252cf" exitCode=0 Sep 10 15:28:36.160670 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:36.160266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bk725" event={"ID":"38c0535a-7a83-4ab9-b450-cb9e4024ab7d","Type":"ContainerDied","Data":"6a555fbafa9683d2b12ddcef02ac9b491ba7ae3c4a9dac334cae836d724252cf"} Sep 10 15:28:36.166201 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:36.163492 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" event={"ID":"62fd9543-387c-4a0e-be77-3662e87014e0","Type":"ContainerStarted","Data":"917f629b85523eb2c0118c076232840c2cc1ec71612bc9b051c2524f7e457226"} Sep 10 15:28:36.166201 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:36.165097 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"6bc7b270fa8dcaa21ce705fbe3d5251737da0007730e96269d8887621103e334"} Sep 10 15:28:36.168929 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:36.168898 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"a724ae05e895dbd70102d6afd37895be533f74f51597a9cd4ed39f2f96d0ea91"} Sep 10 15:28:36.203641 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:36.203590 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dnt8m" podStartSLOduration=3.121945214 podStartE2EDuration="4.203573549s" totalImagesPullingTime="1.081628335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:36.20260865 +0000 UTC m=+85.985821715" watchObservedRunningTime="2026-09-10 15:28:36.203573549 +0000 UTC m=+85.986786614" Sep 10 15:28:37.176024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.175972 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bk725" event={"ID":"38c0535a-7a83-4ab9-b450-cb9e4024ab7d","Type":"ContainerStarted","Data":"9ebb582c17b491880d8e89a7e5c4cd32d2f2d8a269c8569c8197a6429ef1298d"} Sep 10 15:28:37.176405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.176048 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bk725" event={"ID":"38c0535a-7a83-4ab9-b450-cb9e4024ab7d","Type":"ContainerStarted","Data":"5e756d7834768a94d702aa97e01468e33d01ef3cabacef5a49c9090761edbb7f"} Sep 10 15:28:37.178253 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.178220 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8" exitCode=0 Sep 10 15:28:37.178385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.178257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8"} Sep 10 15:28:37.197614 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.197564 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bk725" podStartSLOduration=3.7048061690000003 podStartE2EDuration="5.197547049s" totalImagesPullingTime="1.428853518s" totalInitContainerRuntime="63.887362ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:37.196023019 +0000 UTC m=+86.979236086" watchObservedRunningTime="2026-09-10 15:28:37.197547049 +0000 UTC m=+86.980760114" Sep 10 15:28:37.274856 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.274831 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dc458cb77-lx6jn"] Sep 10 15:28:37.278213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.278196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.280627 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.280594 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 15:28:37.280856 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.280843 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f7igdnjpiv2i6\"" Sep 10 15:28:37.280942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.280923 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-95vgz\"" Sep 10 15:28:37.281034 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.280991 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 15:28:37.281123 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.281051 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 15:28:37.281123 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.281057 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 15:28:37.290887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.290865 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc458cb77-lx6jn"] Sep 10 15:28:37.368203 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-metrics-server-audit-profiles\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.368431 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-secret-metrics-server-client-certs\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.368431 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.368431 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-secret-metrics-server-tls\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.368431 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-audit-log\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.368623 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-client-ca-bundle\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.368623 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.368456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvsb\" (UniqueName: \"kubernetes.io/projected/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-kube-api-access-mkvsb\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469539 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-secret-metrics-server-tls\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469539 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-audit-log\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469539 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-client-ca-bundle\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469826 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvsb\" (UniqueName: \"kubernetes.io/projected/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-kube-api-access-mkvsb\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469826 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-metrics-server-audit-profiles\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469826 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-secret-metrics-server-client-certs\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.469826 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.469674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.470281 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.470255 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-audit-log\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.470402 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.470384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.471010 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.470959 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-metrics-server-audit-profiles\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.472552 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.472531 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-secret-metrics-server-tls\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.472743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.472719 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-secret-metrics-server-client-certs\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.473010 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.472974 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-client-ca-bundle\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.479788 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.479762 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvsb\" (UniqueName: \"kubernetes.io/projected/681f98e0-eacc-4c83-baf5-1bc8f4bbf488-kube-api-access-mkvsb\") pod \"metrics-server-dc458cb77-lx6jn\" (UID: \"681f98e0-eacc-4c83-baf5-1bc8f4bbf488\") " pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.587408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.587376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:37.602978 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.602954 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr"] Sep 10 15:28:37.606210 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.606193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:37.608414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.608393 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 15:28:37.608517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.608418 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b6z2z\"" Sep 10 15:28:37.620315 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.620293 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr"] Sep 10 15:28:37.671688 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.671658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:37.671881 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.671736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvkx\" (UniqueName: \"kubernetes.io/projected/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-kube-api-access-mnvkx\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:37.772927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.772853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvkx\" (UniqueName: \"kubernetes.io/projected/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-kube-api-access-mnvkx\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:37.773120 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.772943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:37.773120 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:37.773082 2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 15:28:37.773215 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:28:37.773145 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-monitoring-plugin-cert podName:bd928188-a6b8-40fe-8a34-3497dd8ecdc2 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:38.27312945 +0000 UTC m=+88.056342493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-t2jfr" (UID: "bd928188-a6b8-40fe-8a34-3497dd8ecdc2") : secret "monitoring-plugin-cert" not found Sep 10 15:28:37.785186 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:37.785162 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvkx\" (UniqueName: \"kubernetes.io/projected/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-kube-api-access-mnvkx\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:38.190938 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.190894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"a58ec1cce0c1246f6fcd4c83c9fdc9e11b63f30803e58f682172421b5dade024"} Sep 10 15:28:38.191303 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.191037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"d2dbd8b9136c382d5725b75edbd05dd34a61e7cd79414f0746b2f49817211ce3"} Sep 10 15:28:38.191303 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.191084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"4049cbcc1d82b37d03f10dffbd58e68628702d955391fd24d59584f53b6a00a2"} Sep 10 15:28:38.198800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.198769 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc458cb77-lx6jn"] Sep 10 15:28:38.202483 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:38.202452 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681f98e0_eacc_4c83_baf5_1bc8f4bbf488.slice/crio-01ca2b35cafbe092340586e1782d309d8d2d121ff24eb4ba4a0a8f430015f373 WatchSource:0}: Error finding container 01ca2b35cafbe092340586e1782d309d8d2d121ff24eb4ba4a0a8f430015f373: Status 404 returned error can't find the container with id 01ca2b35cafbe092340586e1782d309d8d2d121ff24eb4ba4a0a8f430015f373 Sep 10 15:28:38.278645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.278562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:38.281186 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.281160 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd928188-a6b8-40fe-8a34-3497dd8ecdc2-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t2jfr\" (UID: \"bd928188-a6b8-40fe-8a34-3497dd8ecdc2\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:38.516849 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.516815 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:38.801608 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:38.801431 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr"] Sep 10 15:28:39.015115 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:39.015047 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd928188_a6b8_40fe_8a34_3497dd8ecdc2.slice/crio-d58e0fa7b4b9aaafc115e37110a38eb5b2f26f03a83cfc815fe07ca15bfe0e57 WatchSource:0}: Error finding container d58e0fa7b4b9aaafc115e37110a38eb5b2f26f03a83cfc815fe07ca15bfe0e57: Status 404 returned error can't find the container with id d58e0fa7b4b9aaafc115e37110a38eb5b2f26f03a83cfc815fe07ca15bfe0e57 Sep 10 15:28:39.199890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:39.199846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" event={"ID":"bd928188-a6b8-40fe-8a34-3497dd8ecdc2","Type":"ContainerStarted","Data":"d58e0fa7b4b9aaafc115e37110a38eb5b2f26f03a83cfc815fe07ca15bfe0e57"} Sep 10 15:28:39.204289 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:39.204252 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" event={"ID":"681f98e0-eacc-4c83-baf5-1bc8f4bbf488","Type":"ContainerStarted","Data":"01ca2b35cafbe092340586e1782d309d8d2d121ff24eb4ba4a0a8f430015f373"} Sep 10 15:28:40.212489 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.212449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101"} Sep 10 15:28:40.212913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.212539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e"} Sep 10 15:28:40.212913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.212580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a"} Sep 10 15:28:40.212913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.212622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a"} Sep 10 15:28:40.212913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.212658 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442"} Sep 10 15:28:40.212913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.212692 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerStarted","Data":"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030"} Sep 10 15:28:40.215878 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.215846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"e3c69ef05fda06138bc950fd714c49312fbf144ac6936e2a5ff83967b5e0d2b0"} Sep 10 15:28:40.216028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.215913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"2b5a17af3735fff70912233baf11cc3946977a52ee858f1ef35b7fa5c2d26747"} Sep 10 15:28:40.216028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.215945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" event={"ID":"ec489482-f04f-4991-9443-4b64fda06fe2","Type":"ContainerStarted","Data":"98f63b7cdc4f0e153c70b130f0f88285f641c240a9c1cfc12ec1a21a30a82dde"} Sep 10 15:28:40.216028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.215988 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:40.241362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.241309 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.080802546 podStartE2EDuration="7.241294887s" totalImagesPullingTime="3.139009415s" totalInitContainerRuntime="21.482926ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:40.240149815 +0000 UTC m=+90.023362876" watchObservedRunningTime="2026-09-10 15:28:40.241294887 +0000 UTC m=+90.024507952" Sep 10 15:28:40.267572 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:40.267505 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" podStartSLOduration=2.7586273329999997 podStartE2EDuration="6.267486322s" totalImagesPullingTime="3.508858989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:40.267385697 +0000 UTC m=+90.050598762" watchObservedRunningTime="2026-09-10 15:28:40.267486322 +0000 UTC m=+90.050699385" Sep 10 15:28:41.221561 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:41.221523 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" event={"ID":"681f98e0-eacc-4c83-baf5-1bc8f4bbf488","Type":"ContainerStarted","Data":"53f76ada4aaf0bf8fdf2df6042771c0825881c568612eb5f159c69ee7a043aba"} Sep 10 15:28:41.223050 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:41.223018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" event={"ID":"bd928188-a6b8-40fe-8a34-3497dd8ecdc2","Type":"ContainerStarted","Data":"5406dd153d35c72fe07d7b670b3911e07c728dfd2a706f0b91f2491c417f096e"} Sep 10 15:28:41.223731 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:41.223698 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:41.228613 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:41.228597 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" Sep 10 15:28:41.241454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:41.241414 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" podStartSLOduration=1.821674137 podStartE2EDuration="4.241401341s" totalImagesPullingTime="2.419727204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:41.240208752 +0000 UTC m=+91.023421816" watchObservedRunningTime="2026-09-10 15:28:41.241401341 +0000 UTC m=+91.024614435" Sep 10 15:28:41.258099 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:41.258062 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t2jfr" podStartSLOduration=2.647926363 podStartE2EDuration="4.258052822s" totalImagesPullingTime="1.610126459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:41.257261666 +0000 UTC m=+91.040474731" watchObservedRunningTime="2026-09-10 15:28:41.258052822 +0000 UTC m=+91.041265885" Sep 10 15:28:45.773171 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.773137 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-4w9vm"] Sep 10 15:28:45.777601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.777577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:28:45.779948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.779924 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 15:28:45.780094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.779971 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mplvs\"" Sep 10 15:28:45.780094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.779983 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 15:28:45.787104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.787082 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-4w9vm"] Sep 10 15:28:45.947332 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:45.947293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5rn\" (UniqueName: \"kubernetes.io/projected/eac25259-b07b-4e50-af0b-f886e275803e-kube-api-access-6t5rn\") pod \"downloads-85d98c998f-4w9vm\" (UID: \"eac25259-b07b-4e50-af0b-f886e275803e\") " pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:28:46.048753 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:46.048721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5rn\" (UniqueName: \"kubernetes.io/projected/eac25259-b07b-4e50-af0b-f886e275803e-kube-api-access-6t5rn\") pod \"downloads-85d98c998f-4w9vm\" (UID: \"eac25259-b07b-4e50-af0b-f886e275803e\") " pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:28:46.057572 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:46.057545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5rn\" (UniqueName: \"kubernetes.io/projected/eac25259-b07b-4e50-af0b-f886e275803e-kube-api-access-6t5rn\") pod \"downloads-85d98c998f-4w9vm\" (UID: \"eac25259-b07b-4e50-af0b-f886e275803e\") " pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:28:46.086977 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:46.086946 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:28:46.112807 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:46.112783 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68cc9cc884-t9l6k" Sep 10 15:28:46.229132 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:46.229104 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9d5bdb4b6-gj4tw" Sep 10 15:28:46.358709 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:46.358686 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-4w9vm"] Sep 10 15:28:46.361113 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:28:46.361086 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac25259_b07b_4e50_af0b_f886e275803e.slice/crio-19a04572777a5042ccd79eb3eab1354f31005db99a45e4323a29a3a04ed641d6 WatchSource:0}: Error finding container 19a04572777a5042ccd79eb3eab1354f31005db99a45e4323a29a3a04ed641d6: Status 404 returned error can't find the container with id 19a04572777a5042ccd79eb3eab1354f31005db99a45e4323a29a3a04ed641d6 Sep 10 15:28:47.254959 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:47.254916 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-4w9vm" event={"ID":"eac25259-b07b-4e50-af0b-f886e275803e","Type":"ContainerStarted","Data":"19a04572777a5042ccd79eb3eab1354f31005db99a45e4323a29a3a04ed641d6"} Sep 10 15:28:53.089833 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:53.089797 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q5rkh" Sep 10 15:28:56.478458 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.478409 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d8f744f97-hch5b"] Sep 10 15:28:56.481741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.481717 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.484284 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.484266 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 15:28:56.485283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.485257 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 15:28:56.485404 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.485253 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 15:28:56.485404 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.485255 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 15:28:56.485713 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.485696 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-57xrq\"" Sep 10 15:28:56.485777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.485760 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 15:28:56.491057 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.491006 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d8f744f97-hch5b"] Sep 10 15:28:56.550131 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.550096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-oauth-serving-cert\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.550131 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.550135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-oauth-config\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.550345 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.550152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-config\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.550345 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.550258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-serving-cert\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.550345 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.550317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfmv\" (UniqueName: \"kubernetes.io/projected/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-kube-api-access-ssfmv\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.550473 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.550450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-service-ca\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.651553 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.651516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-service-ca\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.651553 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.651567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-oauth-serving-cert\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.651805 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.651595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-oauth-config\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.651805 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.651616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-config\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.651805 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.651668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-serving-cert\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.651805 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.651726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfmv\" (UniqueName: \"kubernetes.io/projected/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-kube-api-access-ssfmv\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.652453 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.652424 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-service-ca\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.652680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.652657 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-oauth-serving-cert\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.652853 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.652835 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-config\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.654689 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.654666 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-serving-cert\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.654795 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.654746 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-oauth-config\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.661637 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.661594 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfmv\" (UniqueName: \"kubernetes.io/projected/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-kube-api-access-ssfmv\") pod \"console-7d8f744f97-hch5b\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:56.793299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:56.793216 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:28:57.587989 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:57.587949 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:28:57.588428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:28:57.588045 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:29:03.328966 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:03.328926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-4w9vm" event={"ID":"eac25259-b07b-4e50-af0b-f886e275803e","Type":"ContainerStarted","Data":"edf12913fa4db078f9c97702c94e7d537c66974c9d736066d151313e787187e8"} Sep 10 15:29:03.329392 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:03.329113 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:29:03.330440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:03.330420 2581 patch_prober.go:28] interesting pod/downloads-85d98c998f-4w9vm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.21:8080/\": dial tcp 10.134.0.21:8080: connect: connection refused" start-of-body= Sep 10 15:29:03.330496 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:03.330463 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-4w9vm" podUID="eac25259-b07b-4e50-af0b-f886e275803e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.21:8080/\": dial tcp 10.134.0.21:8080: connect: connection refused" Sep 10 15:29:03.348835 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:03.348756 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-4w9vm" podStartSLOduration=1.521238418 podStartE2EDuration="18.348736287s" totalImagesPullingTime="16.827497869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:03.346597135 +0000 UTC m=+113.129810200" watchObservedRunningTime="2026-09-10 15:29:03.348736287 +0000 UTC m=+113.131949352" Sep 10 15:29:03.376631 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:03.376514 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d8f744f97-hch5b"] Sep 10 15:29:03.379131 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:29:03.379101 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4636a27a_e42b_4ccc_ae71_4ed5c59b1ccd.slice/crio-4922786ba87a168b24660e6b93540904799d555dba3d449e522b3e8ba5511f5a WatchSource:0}: Error finding container 4922786ba87a168b24660e6b93540904799d555dba3d449e522b3e8ba5511f5a: Status 404 returned error can't find the container with id 4922786ba87a168b24660e6b93540904799d555dba3d449e522b3e8ba5511f5a Sep 10 15:29:04.335608 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:04.335570 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8f744f97-hch5b" event={"ID":"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd","Type":"ContainerStarted","Data":"4922786ba87a168b24660e6b93540904799d555dba3d449e522b3e8ba5511f5a"} Sep 10 15:29:04.343112 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:04.343081 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-4w9vm" Sep 10 15:29:07.353947 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:07.352546 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8f744f97-hch5b" event={"ID":"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd","Type":"ContainerStarted","Data":"a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3"} Sep 10 15:29:07.375502 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:07.375442 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d8f744f97-hch5b" podStartSLOduration=7.6972489920000005 podStartE2EDuration="11.375422452s" totalImagesPullingTime="3.67817346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:07.373140649 +0000 UTC m=+117.156353714" watchObservedRunningTime="2026-09-10 15:29:07.375422452 +0000 UTC m=+117.158635518" Sep 10 15:29:16.390948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.390911 2581 generic.go:356] "Generic (PLEG): container finished" podID="2e1d53ff-0078-41fe-9d4e-66c0629657a1" containerID="6ffa94847a01d24065d2f2a3e71a0ff86d28ff631c25632ca6dafc3505c277f1" exitCode=0 Sep 10 15:29:16.391362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.390984 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" event={"ID":"2e1d53ff-0078-41fe-9d4e-66c0629657a1","Type":"ContainerDied","Data":"6ffa94847a01d24065d2f2a3e71a0ff86d28ff631c25632ca6dafc3505c277f1"} Sep 10 15:29:16.391404 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.391362 2581 scope.go:117] "RemoveContainer" containerID="6ffa94847a01d24065d2f2a3e71a0ff86d28ff631c25632ca6dafc3505c277f1" Sep 10 15:29:16.485964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.485937 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d8f744f97-hch5b"] Sep 10 15:29:16.794557 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.794530 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:29:16.821742 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.821720 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/init-config-reloader/0.log" Sep 10 15:29:16.829499 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.829479 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/alertmanager/0.log" Sep 10 15:29:16.953055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:16.953030 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/config-reloader/0.log" Sep 10 15:29:17.150498 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.150429 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/kube-rbac-proxy-web/0.log" Sep 10 15:29:17.350747 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.350718 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/kube-rbac-proxy/0.log" Sep 10 15:29:17.396782 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.396746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-59q64" event={"ID":"2e1d53ff-0078-41fe-9d4e-66c0629657a1","Type":"ContainerStarted","Data":"6ab2f2327cd436fd11bc1526b9b7252bc3a32b6169503e33826c090704638b75"} Sep 10 15:29:17.549965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.549919 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/kube-rbac-proxy-metric/0.log" Sep 10 15:29:17.593354 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.593325 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:29:17.597221 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.597200 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dc458cb77-lx6jn" Sep 10 15:29:17.750566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:17.750537 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb689dca-0e29-4f7e-abd1-317b55e6a908/prom-label-proxy/0.log" Sep 10 15:29:18.750166 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:18.750136 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dc458cb77-lx6jn_681f98e0-eacc-4c83-baf5-1bc8f4bbf488/metrics-server/0.log" Sep 10 15:29:18.950092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:18.950063 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-t2jfr_bd928188-a6b8-40fe-8a34-3497dd8ecdc2/monitoring-plugin/0.log" Sep 10 15:29:19.149757 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:19.149734 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bk725_38c0535a-7a83-4ab9-b450-cb9e4024ab7d/init-textfile/0.log" Sep 10 15:29:19.351083 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:19.351056 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bk725_38c0535a-7a83-4ab9-b450-cb9e4024ab7d/node-exporter/0.log" Sep 10 15:29:19.550444 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:19.550419 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bk725_38c0535a-7a83-4ab9-b450-cb9e4024ab7d/kube-rbac-proxy/0.log" Sep 10 15:29:20.573840 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:20.573805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:29:20.576043 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:20.576026 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fb2ac2-2577-4d59-ad17-cb67bd293b36-metrics-certs\") pod \"network-metrics-daemon-9rlrq\" (UID: \"e9fb2ac2-2577-4d59-ad17-cb67bd293b36\") " pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:29:20.717462 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:20.717429 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8jcj\"" Sep 10 15:29:20.726224 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:20.726194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9rlrq" Sep 10 15:29:20.950169 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:20.950140 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dnt8m_62fd9543-387c-4a0e-be77-3662e87014e0/kube-rbac-proxy-main/0.log" Sep 10 15:29:21.044505 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:21.044482 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9rlrq"] Sep 10 15:29:21.046951 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:29:21.046917 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fb2ac2_2577_4d59_ad17_cb67bd293b36.slice/crio-42c5b230db9a4ab56a0c5ec81d26c2163f562b0bdb7cf3110253b42dc2037ae7 WatchSource:0}: Error finding container 42c5b230db9a4ab56a0c5ec81d26c2163f562b0bdb7cf3110253b42dc2037ae7: Status 404 returned error can't find the container with id 42c5b230db9a4ab56a0c5ec81d26c2163f562b0bdb7cf3110253b42dc2037ae7 Sep 10 15:29:21.150724 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:21.150662 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dnt8m_62fd9543-387c-4a0e-be77-3662e87014e0/kube-rbac-proxy-self/0.log" Sep 10 15:29:21.350620 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:21.350592 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dnt8m_62fd9543-387c-4a0e-be77-3662e87014e0/openshift-state-metrics/0.log" Sep 10 15:29:21.413121 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:21.413084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9rlrq" event={"ID":"e9fb2ac2-2577-4d59-ad17-cb67bd293b36","Type":"ContainerStarted","Data":"42c5b230db9a4ab56a0c5ec81d26c2163f562b0bdb7cf3110253b42dc2037ae7"} Sep 10 15:29:22.951681 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:22.951645 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-njh9m_71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9/prometheus-operator/0.log" Sep 10 15:29:23.150824 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.150790 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-njh9m_71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9/kube-rbac-proxy/0.log" Sep 10 15:29:23.350103 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.350071 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-nms9x_ea1fe005-48d7-4140-9d69-4e6873fc7c49/prometheus-operator-admission-webhook/0.log" Sep 10 15:29:23.422406 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.422320 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9rlrq" event={"ID":"e9fb2ac2-2577-4d59-ad17-cb67bd293b36","Type":"ContainerStarted","Data":"1eb952b0ea196a635fa6e6a9f077bdaed3279a2f5319a96ec342f51309550689"} Sep 10 15:29:23.422542 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.422409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9rlrq" event={"ID":"e9fb2ac2-2577-4d59-ad17-cb67bd293b36","Type":"ContainerStarted","Data":"4ab280bae4b46993ba73731d3a06113cae05fd6eb9b9a6cb61226362f938af66"} Sep 10 15:29:23.440409 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.440359 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9rlrq" podStartSLOduration=131.984618608 podStartE2EDuration="2m13.440346134s" totalImagesPullingTime="1.455727526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:23.440184073 +0000 UTC m=+133.223397137" watchObservedRunningTime="2026-09-10 15:29:23.440346134 +0000 UTC m=+133.223559199" Sep 10 15:29:23.549501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.549475 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/thanos-query/0.log" Sep 10 15:29:23.749863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.749794 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy-web/0.log" Sep 10 15:29:23.950091 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:23.950063 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy/0.log" Sep 10 15:29:24.150108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:24.150080 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/prom-label-proxy/0.log" Sep 10 15:29:24.349711 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:24.349689 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy-rules/0.log" Sep 10 15:29:24.550708 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:24.550680 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy-metrics/0.log" Sep 10 15:29:24.749966 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:24.749937 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-ch5hg_a3665a98-3024-4e1a-bf29-ac220d83c587/networking-console-plugin/0.log" Sep 10 15:29:25.350328 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:25.350303 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d8f744f97-hch5b_4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd/console/0.log" Sep 10 15:29:25.751338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:25.751259 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-4w9vm_eac25259-b07b-4e50-af0b-f886e275803e/download-server/0.log" Sep 10 15:29:41.508448 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.508381 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d8f744f97-hch5b" podUID="4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" containerName="console" containerID="cri-o://a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3" gracePeriod=15 Sep 10 15:29:41.782783 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.782717 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:29:41.850703 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.850673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-service-ca\") pod \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " Sep 10 15:29:41.850887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.850718 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-config\") pod \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " Sep 10 15:29:41.850887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.850738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-serving-cert\") pod \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " Sep 10 15:29:41.850887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.850864 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-oauth-config\") pod \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " Sep 10 15:29:41.851084 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.850897 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssfmv\" (UniqueName: \"kubernetes.io/projected/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-kube-api-access-ssfmv\") pod \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " Sep 10 15:29:41.851084 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.850966 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-oauth-serving-cert\") pod \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\" (UID: \"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd\") " Sep 10 15:29:41.851197 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.851176 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-service-ca" (OuterVolumeSpecName: "service-ca") pod "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" (UID: "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:29:41.851335 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.851310 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:41.851407 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.851227 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-config" (OuterVolumeSpecName: "console-config") pod "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" (UID: "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:29:41.851407 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.851379 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" (UID: "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:29:41.853174 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.853143 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" (UID: "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:41.853174 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.853160 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" (UID: "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:41.853174 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.853160 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-kube-api-access-ssfmv" (OuterVolumeSpecName: "kube-api-access-ssfmv") pod "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" (UID: "4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd"). InnerVolumeSpecName "kube-api-access-ssfmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:29:41.951874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.951833 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:41.951874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.951866 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:41.951874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.951876 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-console-oauth-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:41.951874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.951886 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfmv\" (UniqueName: \"kubernetes.io/projected/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-kube-api-access-ssfmv\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:41.952147 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:41.951895 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd-oauth-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:42.502668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.502630 2581 generic.go:356] "Generic (PLEG): container finished" podID="4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" containerID="a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3" exitCode=0 Sep 10 15:29:42.502841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.502675 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d8f744f97-hch5b" Sep 10 15:29:42.502841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.502710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8f744f97-hch5b" event={"ID":"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd","Type":"ContainerDied","Data":"a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3"} Sep 10 15:29:42.502841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.502765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d8f744f97-hch5b" event={"ID":"4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd","Type":"ContainerDied","Data":"4922786ba87a168b24660e6b93540904799d555dba3d449e522b3e8ba5511f5a"} Sep 10 15:29:42.502841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.502793 2581 scope.go:117] "RemoveContainer" containerID="a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3" Sep 10 15:29:42.515549 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.515511 2581 scope.go:117] "RemoveContainer" containerID="a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3" Sep 10 15:29:42.515812 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:42.515791 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3\": container with ID starting with a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3 not found: ID does not exist" containerID="a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3" Sep 10 15:29:42.515855 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.515820 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3"} err="failed to get container status \"a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3\": rpc error: code = NotFound desc = could not find container \"a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3\": container with ID starting with a91318c66b7c9abc2536c430a26ec5487131161714092899463f8d83bae317d3 not found: ID does not exist" Sep 10 15:29:42.528062 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.528040 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d8f744f97-hch5b"] Sep 10 15:29:42.532952 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.532932 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d8f744f97-hch5b"] Sep 10 15:29:42.787690 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:42.787611 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" path="/var/lib/kubelet/pods/4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd/volumes" Sep 10 15:29:53.196007 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.195967 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:29:53.196483 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.196247 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="alertmanager" containerID="cri-o://04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030" gracePeriod=120 Sep 10 15:29:53.196483 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.196330 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-metric" containerID="cri-o://d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e" gracePeriod=120 Sep 10 15:29:53.196483 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.196364 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy" containerID="cri-o://1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a" gracePeriod=120 Sep 10 15:29:53.196483 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.196416 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="prom-label-proxy" containerID="cri-o://9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101" gracePeriod=120 Sep 10 15:29:53.196483 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.196366 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="config-reloader" containerID="cri-o://581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442" gracePeriod=120 Sep 10 15:29:53.196753 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.196347 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-web" containerID="cri-o://ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a" gracePeriod=120 Sep 10 15:29:53.549191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549161 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101" exitCode=0 Sep 10 15:29:53.549191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549185 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a" exitCode=0 Sep 10 15:29:53.549191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549192 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442" exitCode=0 Sep 10 15:29:53.549191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549199 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030" exitCode=0 Sep 10 15:29:53.549440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549233 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101"} Sep 10 15:29:53.549440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549295 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a"} Sep 10 15:29:53.549440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442"} Sep 10 15:29:53.549440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:53.549370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030"} Sep 10 15:29:54.449055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.449032 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.553391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553365 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553573 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553408 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-web\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553573 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553437 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-web-config\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553580 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-volume\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553619 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-metrics-client-ca\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553659 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrbgk\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-kube-api-access-wrbgk\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553714 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-main-db\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553742 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-trusted-ca-bundle\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553777 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-out\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-cluster-tls-config\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.553861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553853 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.554129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553899 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.554129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.553928 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-tls-assets\") pod \"fb689dca-0e29-4f7e-abd1-317b55e6a908\" (UID: \"fb689dca-0e29-4f7e-abd1-317b55e6a908\") " Sep 10 15:29:54.554283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.554255 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:29:54.555714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.555680 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:29:54.555841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.555814 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:29:54.556751 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.556615 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-kube-api-access-wrbgk" (OuterVolumeSpecName: "kube-api-access-wrbgk") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "kube-api-access-wrbgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:29:54.557041 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.557017 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-volume" (OuterVolumeSpecName: "config-volume") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.557195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.557063 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.557715 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.557677 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-out" (OuterVolumeSpecName: "config-out") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:29:54.558025 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.557968 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:29:54.558249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558206 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.558249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558235 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.558423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558249 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.558859 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558831 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e" exitCode=0 Sep 10 15:29:54.558948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558861 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerID="ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a" exitCode=0 Sep 10 15:29:54.558948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e"} Sep 10 15:29:54.558948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558915 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.558948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558944 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a"} Sep 10 15:29:54.559122 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.558987 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb689dca-0e29-4f7e-abd1-317b55e6a908","Type":"ContainerDied","Data":"6bc7b270fa8dcaa21ce705fbe3d5251737da0007730e96269d8887621103e334"} Sep 10 15:29:54.559122 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.559034 2581 scope.go:117] "RemoveContainer" containerID="9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101" Sep 10 15:29:54.562618 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.562510 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.568736 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.568716 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-web-config" (OuterVolumeSpecName: "web-config") pod "fb689dca-0e29-4f7e-abd1-317b55e6a908" (UID: "fb689dca-0e29-4f7e-abd1-317b55e6a908"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:29:54.570235 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.570219 2581 scope.go:117] "RemoveContainer" containerID="d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e" Sep 10 15:29:54.576900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.576884 2581 scope.go:117] "RemoveContainer" containerID="1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a" Sep 10 15:29:54.583309 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.583290 2581 scope.go:117] "RemoveContainer" containerID="ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a" Sep 10 15:29:54.589707 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.589691 2581 scope.go:117] "RemoveContainer" containerID="581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442" Sep 10 15:29:54.596436 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.596415 2581 scope.go:117] "RemoveContainer" containerID="04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030" Sep 10 15:29:54.603272 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.603255 2581 scope.go:117] "RemoveContainer" containerID="eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8" Sep 10 15:29:54.610035 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610018 2581 scope.go:117] "RemoveContainer" containerID="9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101" Sep 10 15:29:54.610294 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.610276 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101\": container with ID starting with 9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101 not found: ID does not exist" containerID="9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101" Sep 10 15:29:54.610338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610305 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101"} err="failed to get container status \"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101\": rpc error: code = NotFound desc = could not find container \"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101\": container with ID starting with 9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101 not found: ID does not exist" Sep 10 15:29:54.610338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610323 2581 scope.go:117] "RemoveContainer" containerID="d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e" Sep 10 15:29:54.610579 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.610564 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e\": container with ID starting with d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e not found: ID does not exist" containerID="d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e" Sep 10 15:29:54.610638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610582 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e"} err="failed to get container status \"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e\": rpc error: code = NotFound desc = could not find container \"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e\": container with ID starting with d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e not found: ID does not exist" Sep 10 15:29:54.610638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610595 2581 scope.go:117] "RemoveContainer" containerID="1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a" Sep 10 15:29:54.610836 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.610819 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a\": container with ID starting with 1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a not found: ID does not exist" containerID="1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a" Sep 10 15:29:54.610878 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610841 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a"} err="failed to get container status \"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a\": rpc error: code = NotFound desc = could not find container \"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a\": container with ID starting with 1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a not found: ID does not exist" Sep 10 15:29:54.610878 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.610857 2581 scope.go:117] "RemoveContainer" containerID="ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a" Sep 10 15:29:54.611116 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.611100 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a\": container with ID starting with ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a not found: ID does not exist" containerID="ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a" Sep 10 15:29:54.611165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611120 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a"} err="failed to get container status \"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a\": rpc error: code = NotFound desc = could not find container \"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a\": container with ID starting with ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a not found: ID does not exist" Sep 10 15:29:54.611165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611135 2581 scope.go:117] "RemoveContainer" containerID="581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442" Sep 10 15:29:54.611376 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.611360 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442\": container with ID starting with 581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442 not found: ID does not exist" containerID="581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442" Sep 10 15:29:54.611415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611382 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442"} err="failed to get container status \"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442\": rpc error: code = NotFound desc = could not find container \"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442\": container with ID starting with 581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442 not found: ID does not exist" Sep 10 15:29:54.611415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611396 2581 scope.go:117] "RemoveContainer" containerID="04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030" Sep 10 15:29:54.611629 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.611611 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030\": container with ID starting with 04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030 not found: ID does not exist" containerID="04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030" Sep 10 15:29:54.611680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611632 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030"} err="failed to get container status \"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030\": rpc error: code = NotFound desc = could not find container \"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030\": container with ID starting with 04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030 not found: ID does not exist" Sep 10 15:29:54.611680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611647 2581 scope.go:117] "RemoveContainer" containerID="eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8" Sep 10 15:29:54.611876 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:29:54.611860 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8\": container with ID starting with eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8 not found: ID does not exist" containerID="eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8" Sep 10 15:29:54.611918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611878 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8"} err="failed to get container status \"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8\": rpc error: code = NotFound desc = could not find container \"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8\": container with ID starting with eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8 not found: ID does not exist" Sep 10 15:29:54.611918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.611896 2581 scope.go:117] "RemoveContainer" containerID="9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101" Sep 10 15:29:54.612154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612135 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101"} err="failed to get container status \"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101\": rpc error: code = NotFound desc = could not find container \"9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101\": container with ID starting with 9f02f6bcd078b2b2acf300cb11fef55e9e74a5ed668c0261a3e149c7ae349101 not found: ID does not exist" Sep 10 15:29:54.612226 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612156 2581 scope.go:117] "RemoveContainer" containerID="d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e" Sep 10 15:29:54.612373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612359 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e"} err="failed to get container status \"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e\": rpc error: code = NotFound desc = could not find container \"d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e\": container with ID starting with d4a91af415e15b2341b8cb21a91751257ae0a28756de5c7b7b490ae8525a096e not found: ID does not exist" Sep 10 15:29:54.612426 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612374 2581 scope.go:117] "RemoveContainer" containerID="1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a" Sep 10 15:29:54.612596 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612578 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a"} err="failed to get container status \"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a\": rpc error: code = NotFound desc = could not find container \"1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a\": container with ID starting with 1c0b6249a35951a128d6742d205bb460d1a8a2766d1e88c0cdd8ca1525a4de7a not found: ID does not exist" Sep 10 15:29:54.612641 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612596 2581 scope.go:117] "RemoveContainer" containerID="ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a" Sep 10 15:29:54.612809 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612792 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a"} err="failed to get container status \"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a\": rpc error: code = NotFound desc = could not find container \"ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a\": container with ID starting with ecbb328760582ea9265d02c8d0751a0e90f8b5b8d46821e2a800d6d3d1aa809a not found: ID does not exist" Sep 10 15:29:54.612850 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.612809 2581 scope.go:117] "RemoveContainer" containerID="581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442" Sep 10 15:29:54.613037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.613017 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442"} err="failed to get container status \"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442\": rpc error: code = NotFound desc = could not find container \"581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442\": container with ID starting with 581a02a9a581d6c034bd6effaae19dfa226d7e867434335956c93e2eb1e0e442 not found: ID does not exist" Sep 10 15:29:54.613110 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.613039 2581 scope.go:117] "RemoveContainer" containerID="04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030" Sep 10 15:29:54.613264 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.613245 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030"} err="failed to get container status \"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030\": rpc error: code = NotFound desc = could not find container \"04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030\": container with ID starting with 04f34b57eee1af393c58cd2e6c559dce90a7cd168a0618a0ef059d6ed1b96030 not found: ID does not exist" Sep 10 15:29:54.613321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.613264 2581 scope.go:117] "RemoveContainer" containerID="eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8" Sep 10 15:29:54.613467 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.613450 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8"} err="failed to get container status \"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8\": rpc error: code = NotFound desc = could not find container \"eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8\": container with ID starting with eb389d85b4055f370ea6693f0ef12336e4e982a29994f19684843ae7c655aef8 not found: ID does not exist" Sep 10 15:29:54.655529 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655502 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-main-db\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655529 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655527 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655541 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-out\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655553 2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-cluster-tls-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655565 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-main-tls\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655581 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655593 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-tls-assets\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655604 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655619 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655632 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-web-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655645 2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb689dca-0e29-4f7e-abd1-317b55e6a908-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655656 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb689dca-0e29-4f7e-abd1-317b55e6a908-metrics-client-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.655705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.655669 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrbgk\" (UniqueName: \"kubernetes.io/projected/fb689dca-0e29-4f7e-abd1-317b55e6a908-kube-api-access-wrbgk\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:29:54.877910 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.877827 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:29:54.884056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.884033 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:29:54.912221 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912192 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:29:54.912478 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912466 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="alertmanager" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912481 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="alertmanager" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912491 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" containerName="console" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912496 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" containerName="console" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912507 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="prom-label-proxy" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912513 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="prom-label-proxy" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912523 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="config-reloader" Sep 10 15:29:54.912530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912529 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="config-reloader" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912534 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912539 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912548 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-web" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912553 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-web" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912560 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-metric" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912566 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-metric" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912572 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="init-config-reloader" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912576 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="init-config-reloader" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912620 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="config-reloader" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912628 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="prom-label-proxy" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912635 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-metric" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912641 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912646 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="alertmanager" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912652 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" containerName="kube-rbac-proxy-web" Sep 10 15:29:54.912800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.912658 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4636a27a-e42b-4ccc-ae71-4ed5c59b1ccd" containerName="console" Sep 10 15:29:54.918415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.918393 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.920809 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.920782 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 15:29:54.920809 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.920795 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 15:29:54.921290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921091 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 15:29:54.921290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921122 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hprwv\"" Sep 10 15:29:54.921290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921175 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 15:29:54.921290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921211 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 15:29:54.921290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921228 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 15:29:54.921671 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921651 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 15:29:54.921789 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.921770 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 15:29:54.925731 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.925709 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 15:29:54.933371 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.933352 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:29:54.957546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-web-config\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957678 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957712 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51063e07-8d94-4186-a742-b689ca1690e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957844 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957761 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51063e07-8d94-4186-a742-b689ca1690e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957844 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51063e07-8d94-4186-a742-b689ca1690e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957844 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51063e07-8d94-4186-a742-b689ca1690e2-config-out\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.957844 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957829 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.958021 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh522\" (UniqueName: \"kubernetes.io/projected/51063e07-8d94-4186-a742-b689ca1690e2-kube-api-access-lh522\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:54.958021 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:54.957877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51063e07-8d94-4186-a742-b689ca1690e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-web-config\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51063e07-8d94-4186-a742-b689ca1690e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058519 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51063e07-8d94-4186-a742-b689ca1690e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51063e07-8d94-4186-a742-b689ca1690e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51063e07-8d94-4186-a742-b689ca1690e2-config-out\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058624 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.058660 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh522\" (UniqueName: \"kubernetes.io/projected/51063e07-8d94-4186-a742-b689ca1690e2-kube-api-access-lh522\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.059129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51063e07-8d94-4186-a742-b689ca1690e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.059129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.058709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.059129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.059041 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51063e07-8d94-4186-a742-b689ca1690e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.059614 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.059587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51063e07-8d94-4186-a742-b689ca1690e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.060753 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.060722 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51063e07-8d94-4186-a742-b689ca1690e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.061790 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.061663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.061790 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.061737 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-web-config\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.061944 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.061807 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.062276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.062258 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.062276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.062270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.062403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.062305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.062403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.062386 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51063e07-8d94-4186-a742-b689ca1690e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.062403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.062395 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51063e07-8d94-4186-a742-b689ca1690e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.063149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.063134 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51063e07-8d94-4186-a742-b689ca1690e2-config-out\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.069081 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.069064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh522\" (UniqueName: \"kubernetes.io/projected/51063e07-8d94-4186-a742-b689ca1690e2-kube-api-access-lh522\") pod \"alertmanager-main-0\" (UID: \"51063e07-8d94-4186-a742-b689ca1690e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.228542 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.228455 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:29:55.504439 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.504216 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:29:55.506983 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:29:55.506957 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51063e07_8d94_4186_a742_b689ca1690e2.slice/crio-ee3d241d7fa6580f455f56724e4d30929142a224c1b1fd95d25ae58e008735db WatchSource:0}: Error finding container ee3d241d7fa6580f455f56724e4d30929142a224c1b1fd95d25ae58e008735db: Status 404 returned error can't find the container with id ee3d241d7fa6580f455f56724e4d30929142a224c1b1fd95d25ae58e008735db Sep 10 15:29:55.565628 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:55.565604 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"ee3d241d7fa6580f455f56724e4d30929142a224c1b1fd95d25ae58e008735db"} Sep 10 15:29:56.573542 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:56.573504 2581 generic.go:356] "Generic (PLEG): container finished" podID="51063e07-8d94-4186-a742-b689ca1690e2" containerID="29e9ad6bca8b8d6c190d1eeb11c8acc1c1d97418b229fe79c51e896a373804ef" exitCode=0 Sep 10 15:29:56.573885 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:56.573598 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerDied","Data":"29e9ad6bca8b8d6c190d1eeb11c8acc1c1d97418b229fe79c51e896a373804ef"} Sep 10 15:29:56.787104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:56.787080 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb689dca-0e29-4f7e-abd1-317b55e6a908" path="/var/lib/kubelet/pods/fb689dca-0e29-4f7e-abd1-317b55e6a908/volumes" Sep 10 15:29:57.230041 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.229941 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq"] Sep 10 15:29:57.233395 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.233371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.235842 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.235810 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 15:29:57.235985 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.235816 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2h96h\"" Sep 10 15:29:57.236078 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.236008 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 15:29:57.236142 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.236092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 15:29:57.236142 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.236101 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 15:29:57.236345 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.236332 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 15:29:57.241585 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.241561 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 15:29:57.249391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.249370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq"] Sep 10 15:29:57.282588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-serving-certs-ca-bundle\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282746 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282746 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-telemeter-client-tls\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmp27\" (UniqueName: \"kubernetes.io/projected/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-kube-api-access-rmp27\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-metrics-client-ca\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282988 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-secret-telemeter-client\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.282988 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.282915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-federate-client-tls\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmp27\" (UniqueName: \"kubernetes.io/projected/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-kube-api-access-rmp27\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-metrics-client-ca\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-secret-telemeter-client\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-federate-client-tls\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-serving-certs-ca-bundle\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.383914 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.383666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-telemeter-client-tls\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.384384 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.384319 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-metrics-client-ca\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.384384 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.384339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-serving-certs-ca-bundle\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.385245 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.385220 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.386525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.386481 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.386661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.386636 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-telemeter-client-tls\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.386812 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.386788 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-secret-telemeter-client\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.387154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.387135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-federate-client-tls\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.392974 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.392948 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmp27\" (UniqueName: \"kubernetes.io/projected/1733fc28-1877-4e80-a9c8-92f3b47a5b8f-kube-api-access-rmp27\") pod \"telemeter-client-58d8cc56c9-cnnrq\" (UID: \"1733fc28-1877-4e80-a9c8-92f3b47a5b8f\") " pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.556626 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.556582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" Sep 10 15:29:57.587237 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.587203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"95e7be1f036c3606be2449d111a215a3d4029bf8341d2921afb5769b94bf2e40"} Sep 10 15:29:57.587657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.587318 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"b5ca0b02d185bf0d6d2c918bcca763d8d7b58bbcf7b969548f3c3858e47e17be"} Sep 10 15:29:57.587657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.587371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"02bc93f64698dbf25bc586ee28d60b7e7ea9073cdc481e56889c030856564358"} Sep 10 15:29:57.587657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.587416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"3feb2e144da1fa9f5e3f9246ec96df5232900a031535862956dc86a9ca2c15f6"} Sep 10 15:29:57.587657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.587456 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"5e8d2b71e7c38337d5e8ca1e46c4fb7e2b46807d905b7f8aa0f5dd5146e8b631"} Sep 10 15:29:57.587657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.587495 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51063e07-8d94-4186-a742-b689ca1690e2","Type":"ContainerStarted","Data":"9844695e79918d4d1e6be72ddfce38bec1a2399f74ae33d8889d08a2f8867074"} Sep 10 15:29:57.614377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.614292 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.594793487 podStartE2EDuration="3.614277555s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.484068ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:57.614091183 +0000 UTC m=+167.397304248" watchObservedRunningTime="2026-09-10 15:29:57.614277555 +0000 UTC m=+167.397490618" Sep 10 15:29:57.854426 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:57.854335 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq"] Sep 10 15:29:57.859040 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:29:57.859008 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1733fc28_1877_4e80_a9c8_92f3b47a5b8f.slice/crio-ade657333f93b146e0ce583afcdc13dc229a0b8b968728d6ddc543223e2a424e WatchSource:0}: Error finding container ade657333f93b146e0ce583afcdc13dc229a0b8b968728d6ddc543223e2a424e: Status 404 returned error can't find the container with id ade657333f93b146e0ce583afcdc13dc229a0b8b968728d6ddc543223e2a424e Sep 10 15:29:58.594684 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:29:58.594651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" event={"ID":"1733fc28-1877-4e80-a9c8-92f3b47a5b8f","Type":"ContainerStarted","Data":"ade657333f93b146e0ce583afcdc13dc229a0b8b968728d6ddc543223e2a424e"} Sep 10 15:30:00.604689 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.604648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" event={"ID":"1733fc28-1877-4e80-a9c8-92f3b47a5b8f","Type":"ContainerStarted","Data":"3e1d023082b154f0275c2b2d7f2ffdeee548f1eb2045025c0ce0553a30a2097a"} Sep 10 15:30:00.605128 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.604721 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" event={"ID":"1733fc28-1877-4e80-a9c8-92f3b47a5b8f","Type":"ContainerStarted","Data":"256f753b9323dc4bc543d97bbc19448fa067380984ea87fc4758b01db1721ed5"} Sep 10 15:30:00.605128 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.604750 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" event={"ID":"1733fc28-1877-4e80-a9c8-92f3b47a5b8f","Type":"ContainerStarted","Data":"1e45b6566ebf67d705e596de05e6a02c3b815036cf7728c5383ed215e66353c8"} Sep 10 15:30:00.628012 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.627946 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58d8cc56c9-cnnrq" podStartSLOduration=1.759185929 podStartE2EDuration="3.627929511s" totalImagesPullingTime="1.868743582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:30:00.626141132 +0000 UTC m=+170.409354195" watchObservedRunningTime="2026-09-10 15:30:00.627929511 +0000 UTC m=+170.411142574" Sep 10 15:30:00.717666 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.717637 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-786996d479-vghjz"] Sep 10 15:30:00.719784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.719768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.723163 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:30:00.723135 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-786996d479-vghjz\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.724102 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:30:00.724074 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"console-config\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" Sep 10 15:30:00.724214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.724106 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 15:30:00.724214 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:30:00.724139 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" Sep 10 15:30:00.724296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.724221 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-57xrq\"" Sep 10 15:30:00.724296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.724277 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 15:30:00.724357 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.724309 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 15:30:00.725512 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:30:00.725490 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"oauth-serving-cert\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" Sep 10 15:30:00.748347 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.748321 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786996d479-vghjz"] Sep 10 15:30:00.812862 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.812830 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-oauth-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.813052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.812874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lf6p\" (UniqueName: \"kubernetes.io/projected/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-kube-api-access-7lf6p\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.813052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.812892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-serving-cert\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.813052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.812949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.813221 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.813074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-trusted-ca-bundle\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.813221 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.813124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-service-ca\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.813221 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.813143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-oauth-serving-cert\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914281 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914192 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf6p\" (UniqueName: \"kubernetes.io/projected/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-kube-api-access-7lf6p\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914281 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-serving-cert\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-trusted-ca-bundle\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-service-ca\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-oauth-serving-cert\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.914645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.914512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-oauth-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.915072 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.915052 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-service-ca\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.916756 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.916736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-serving-cert\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.916841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.916769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-oauth-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:00.923028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:00.922986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lf6p\" (UniqueName: \"kubernetes.io/projected/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-kube-api-access-7lf6p\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:01.723244 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:01.723214 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 15:30:01.725731 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:01.725701 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-trusted-ca-bundle\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:01.791673 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:01.791638 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 15:30:01.795949 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:01.795929 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-oauth-serving-cert\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:01.915567 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:30:01.915521 2581 configmap.go:193] Couldn't get configMap openshift-console/console-config: failed to sync configmap cache: timed out waiting for the condition Sep 10 15:30:01.915749 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:30:01.915615 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config podName:49fec499-64b8-48c1-8f39-d64c5c8d2ebd nodeName:}" failed. No retries permitted until 2026-09-10 15:30:02.415594842 +0000 UTC m=+172.198807894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config") pod "console-786996d479-vghjz" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd") : failed to sync configmap cache: timed out waiting for the condition Sep 10 15:30:02.193792 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:02.193763 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 15:30:02.427849 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:02.427791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:02.428444 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:02.428423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config\") pod \"console-786996d479-vghjz\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:02.528663 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:02.528593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:02.799969 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:02.799935 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786996d479-vghjz"] Sep 10 15:30:02.802875 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:30:02.802851 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fec499_64b8_48c1_8f39_d64c5c8d2ebd.slice/crio-0994c95cf16a9cee0a18b557d39ebe6853353eaeb10b044850ecae128df0e793 WatchSource:0}: Error finding container 0994c95cf16a9cee0a18b557d39ebe6853353eaeb10b044850ecae128df0e793: Status 404 returned error can't find the container with id 0994c95cf16a9cee0a18b557d39ebe6853353eaeb10b044850ecae128df0e793 Sep 10 15:30:03.620276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:03.620244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786996d479-vghjz" event={"ID":"49fec499-64b8-48c1-8f39-d64c5c8d2ebd","Type":"ContainerStarted","Data":"a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269"} Sep 10 15:30:03.620453 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:03.620292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786996d479-vghjz" event={"ID":"49fec499-64b8-48c1-8f39-d64c5c8d2ebd","Type":"ContainerStarted","Data":"0994c95cf16a9cee0a18b557d39ebe6853353eaeb10b044850ecae128df0e793"} Sep 10 15:30:03.638227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:03.638137 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-786996d479-vghjz" podStartSLOduration=3.638122119 podStartE2EDuration="3.638122119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:30:03.637290598 +0000 UTC m=+173.420503664" watchObservedRunningTime="2026-09-10 15:30:03.638122119 +0000 UTC m=+173.421335182" Sep 10 15:30:12.528776 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:12.528737 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:12.529276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:12.528828 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:12.533477 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:12.533454 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:30:12.660415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:30:12.660388 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:31:20.927233 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:20.927200 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786996d479-vghjz"] Sep 10 15:31:45.956248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:45.956146 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-786996d479-vghjz" podUID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" containerName="console" containerID="cri-o://a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269" gracePeriod=15 Sep 10 15:31:46.201775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.201750 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:31:46.313069 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313030 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-trusted-ca-bundle\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313069 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313068 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-service-ca\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313097 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-oauth-config\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313197 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-oauth-serving-cert\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313234 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lf6p\" (UniqueName: \"kubernetes.io/projected/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-kube-api-access-7lf6p\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313335 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-serving-cert\") pod \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\" (UID: \"49fec499-64b8-48c1-8f39-d64c5c8d2ebd\") " Sep 10 15:31:46.313598 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313556 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:31:46.313642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313584 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-service-ca" (OuterVolumeSpecName: "service-ca") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:31:46.313642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313619 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:31:46.313711 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.313641 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config" (OuterVolumeSpecName: "console-config") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:31:46.315302 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.315270 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:31:46.315404 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.315376 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:31:46.315454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.315421 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-kube-api-access-7lf6p" (OuterVolumeSpecName: "kube-api-access-7lf6p") pod "49fec499-64b8-48c1-8f39-d64c5c8d2ebd" (UID: "49fec499-64b8-48c1-8f39-d64c5c8d2ebd"). InnerVolumeSpecName "kube-api-access-7lf6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:31:46.415055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.414989 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-trusted-ca-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:46.415055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.415046 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:46.415055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.415057 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-oauth-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:46.415055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.415066 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-oauth-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:46.415312 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.415075 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-config\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:46.415312 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.415085 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lf6p\" (UniqueName: \"kubernetes.io/projected/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-kube-api-access-7lf6p\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:46.415312 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:46.415093 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49fec499-64b8-48c1-8f39-d64c5c8d2ebd-console-serving-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:31:47.032242 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.032203 2581 generic.go:356] "Generic (PLEG): container finished" podID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" containerID="a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269" exitCode=0 Sep 10 15:31:47.032719 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.032250 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786996d479-vghjz" Sep 10 15:31:47.032719 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.032285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786996d479-vghjz" event={"ID":"49fec499-64b8-48c1-8f39-d64c5c8d2ebd","Type":"ContainerDied","Data":"a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269"} Sep 10 15:31:47.032719 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.032348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786996d479-vghjz" event={"ID":"49fec499-64b8-48c1-8f39-d64c5c8d2ebd","Type":"ContainerDied","Data":"0994c95cf16a9cee0a18b557d39ebe6853353eaeb10b044850ecae128df0e793"} Sep 10 15:31:47.032719 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.032375 2581 scope.go:117] "RemoveContainer" containerID="a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269" Sep 10 15:31:47.040540 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.040520 2581 scope.go:117] "RemoveContainer" containerID="a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269" Sep 10 15:31:47.040802 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:31:47.040774 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269\": container with ID starting with a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269 not found: ID does not exist" containerID="a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269" Sep 10 15:31:47.040856 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.040803 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269"} err="failed to get container status \"a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269\": rpc error: code = NotFound desc = could not find container \"a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269\": container with ID starting with a9f28ddf97f113324745e397a260a022e2f1b6270b25e604ebb51adec5197269 not found: ID does not exist" Sep 10 15:31:47.063247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.063187 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786996d479-vghjz"] Sep 10 15:31:47.068971 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:47.068949 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-786996d479-vghjz"] Sep 10 15:31:48.787398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:31:48.787361 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" path="/var/lib/kubelet/pods/49fec499-64b8-48c1-8f39-d64c5c8d2ebd/volumes" Sep 10 15:32:10.605555 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:10.605510 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 15:32:11.485153 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.485113 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z"] Sep 10 15:32:11.487195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.485519 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" containerName="console" Sep 10 15:32:11.487195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.485535 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" containerName="console" Sep 10 15:32:11.487195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.485587 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49fec499-64b8-48c1-8f39-d64c5c8d2ebd" containerName="console" Sep 10 15:32:11.488574 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.488554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.491268 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.491242 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:32:11.491377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.491273 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-984bz\"" Sep 10 15:32:11.492053 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.492037 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:32:11.497200 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.497180 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z"] Sep 10 15:32:11.530948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.530916 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtcn\" (UniqueName: \"kubernetes.io/projected/f7e321ad-a5f5-4ddc-93b9-be65d3541490-kube-api-access-vbtcn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.530948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.530949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.531168 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.530980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.632111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.632063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtcn\" (UniqueName: \"kubernetes.io/projected/f7e321ad-a5f5-4ddc-93b9-be65d3541490-kube-api-access-vbtcn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.632111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.632111 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.632519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.632141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.632519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.632457 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.632586 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.632527 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.641703 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.641670 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtcn\" (UniqueName: \"kubernetes.io/projected/f7e321ad-a5f5-4ddc-93b9-be65d3541490-kube-api-access-vbtcn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:11.799062 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:11.799034 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:12.068792 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:12.068728 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z"] Sep 10 15:32:12.071733 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:32:12.071701 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e321ad_a5f5_4ddc_93b9_be65d3541490.slice/crio-1f7799c7f07d4a3a8439feaee722eae20433fb3c85b502c83723d1ceffa4b65d WatchSource:0}: Error finding container 1f7799c7f07d4a3a8439feaee722eae20433fb3c85b502c83723d1ceffa4b65d: Status 404 returned error can't find the container with id 1f7799c7f07d4a3a8439feaee722eae20433fb3c85b502c83723d1ceffa4b65d Sep 10 15:32:12.074111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:12.074095 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:32:12.140503 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:12.140470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" event={"ID":"f7e321ad-a5f5-4ddc-93b9-be65d3541490","Type":"ContainerStarted","Data":"1f7799c7f07d4a3a8439feaee722eae20433fb3c85b502c83723d1ceffa4b65d"} Sep 10 15:32:18.174006 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:18.173956 2581 generic.go:356] "Generic (PLEG): container finished" podID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerID="19b7ecb4bc565738ad2af62c704b8188508175f036803fb2a4d9a28a726e4142" exitCode=0 Sep 10 15:32:18.174403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:18.174039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" event={"ID":"f7e321ad-a5f5-4ddc-93b9-be65d3541490","Type":"ContainerDied","Data":"19b7ecb4bc565738ad2af62c704b8188508175f036803fb2a4d9a28a726e4142"} Sep 10 15:32:20.184481 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:20.184381 2581 generic.go:356] "Generic (PLEG): container finished" podID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerID="fe6342a4a78f294487986918bc818ca1bf3cd8c5bdc317ef448b6bde496b3ae6" exitCode=0 Sep 10 15:32:20.184481 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:20.184438 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" event={"ID":"f7e321ad-a5f5-4ddc-93b9-be65d3541490","Type":"ContainerDied","Data":"fe6342a4a78f294487986918bc818ca1bf3cd8c5bdc317ef448b6bde496b3ae6"} Sep 10 15:32:26.219649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:26.219609 2581 generic.go:356] "Generic (PLEG): container finished" podID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerID="07277503dcf70b6bb0f0a705329515e476a5ee87a4b4a6440dc1f658e86ce4c8" exitCode=0 Sep 10 15:32:26.220021 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:26.219668 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" event={"ID":"f7e321ad-a5f5-4ddc-93b9-be65d3541490","Type":"ContainerDied","Data":"07277503dcf70b6bb0f0a705329515e476a5ee87a4b4a6440dc1f658e86ce4c8"} Sep 10 15:32:27.347416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.347386 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:27.472020 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.471962 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbtcn\" (UniqueName: \"kubernetes.io/projected/f7e321ad-a5f5-4ddc-93b9-be65d3541490-kube-api-access-vbtcn\") pod \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " Sep 10 15:32:27.472020 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.472024 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-bundle\") pod \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " Sep 10 15:32:27.472258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.472050 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-util\") pod \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\" (UID: \"f7e321ad-a5f5-4ddc-93b9-be65d3541490\") " Sep 10 15:32:27.472661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.472630 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-bundle" (OuterVolumeSpecName: "bundle") pod "f7e321ad-a5f5-4ddc-93b9-be65d3541490" (UID: "f7e321ad-a5f5-4ddc-93b9-be65d3541490"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:32:27.474137 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.474114 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e321ad-a5f5-4ddc-93b9-be65d3541490-kube-api-access-vbtcn" (OuterVolumeSpecName: "kube-api-access-vbtcn") pod "f7e321ad-a5f5-4ddc-93b9-be65d3541490" (UID: "f7e321ad-a5f5-4ddc-93b9-be65d3541490"). InnerVolumeSpecName "kube-api-access-vbtcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:32:27.476014 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.475948 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-util" (OuterVolumeSpecName: "util") pod "f7e321ad-a5f5-4ddc-93b9-be65d3541490" (UID: "f7e321ad-a5f5-4ddc-93b9-be65d3541490"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:32:27.572906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.572858 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbtcn\" (UniqueName: \"kubernetes.io/projected/f7e321ad-a5f5-4ddc-93b9-be65d3541490-kube-api-access-vbtcn\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:32:27.572906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.572902 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:32:27.572906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:27.572912 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7e321ad-a5f5-4ddc-93b9-be65d3541490-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:32:28.229301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:28.229266 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" Sep 10 15:32:28.229445 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:28.229268 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggs7z" event={"ID":"f7e321ad-a5f5-4ddc-93b9-be65d3541490","Type":"ContainerDied","Data":"1f7799c7f07d4a3a8439feaee722eae20433fb3c85b502c83723d1ceffa4b65d"} Sep 10 15:32:28.229445 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:28.229385 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f7799c7f07d4a3a8439feaee722eae20433fb3c85b502c83723d1ceffa4b65d" Sep 10 15:32:34.530513 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.530477 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z"] Sep 10 15:32:34.531226 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531205 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="pull" Sep 10 15:32:34.531373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531231 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="pull" Sep 10 15:32:34.531373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531249 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="extract" Sep 10 15:32:34.531373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531257 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="extract" Sep 10 15:32:34.531373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531301 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="util" Sep 10 15:32:34.531373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531310 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="util" Sep 10 15:32:34.531647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.531453 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e321ad-a5f5-4ddc-93b9-be65d3541490" containerName="extract" Sep 10 15:32:34.552701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.552673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.556404 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.556381 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ccbv6\"" Sep 10 15:32:34.561381 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.561363 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:32:34.561469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.561390 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 15:32:34.577465 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.577443 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z"] Sep 10 15:32:34.633668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.633641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85de4b73-b795-455d-835c-17ad8fcaea3d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t689z\" (UID: \"85de4b73-b795-455d-835c-17ad8fcaea3d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.633796 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.633692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdl7\" (UniqueName: \"kubernetes.io/projected/85de4b73-b795-455d-835c-17ad8fcaea3d-kube-api-access-zmdl7\") pod \"cert-manager-operator-controller-manager-55c444bf86-t689z\" (UID: \"85de4b73-b795-455d-835c-17ad8fcaea3d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.734627 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.734586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdl7\" (UniqueName: \"kubernetes.io/projected/85de4b73-b795-455d-835c-17ad8fcaea3d-kube-api-access-zmdl7\") pod \"cert-manager-operator-controller-manager-55c444bf86-t689z\" (UID: \"85de4b73-b795-455d-835c-17ad8fcaea3d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.734790 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.734672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85de4b73-b795-455d-835c-17ad8fcaea3d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t689z\" (UID: \"85de4b73-b795-455d-835c-17ad8fcaea3d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.735049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.735034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85de4b73-b795-455d-835c-17ad8fcaea3d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t689z\" (UID: \"85de4b73-b795-455d-835c-17ad8fcaea3d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.745794 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.745770 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdl7\" (UniqueName: \"kubernetes.io/projected/85de4b73-b795-455d-835c-17ad8fcaea3d-kube-api-access-zmdl7\") pod \"cert-manager-operator-controller-manager-55c444bf86-t689z\" (UID: \"85de4b73-b795-455d-835c-17ad8fcaea3d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:34.861433 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:34.861398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" Sep 10 15:32:35.163942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:35.163918 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z"] Sep 10 15:32:35.165739 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:32:35.165711 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85de4b73_b795_455d_835c_17ad8fcaea3d.slice/crio-0fd2b1e168c11063eb9c015bda8408c4f3803bb1439f826214d4be7c4c40c9f3 WatchSource:0}: Error finding container 0fd2b1e168c11063eb9c015bda8408c4f3803bb1439f826214d4be7c4c40c9f3: Status 404 returned error can't find the container with id 0fd2b1e168c11063eb9c015bda8408c4f3803bb1439f826214d4be7c4c40c9f3 Sep 10 15:32:35.262923 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:35.262888 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" event={"ID":"85de4b73-b795-455d-835c-17ad8fcaea3d","Type":"ContainerStarted","Data":"0fd2b1e168c11063eb9c015bda8408c4f3803bb1439f826214d4be7c4c40c9f3"} Sep 10 15:32:38.153763 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.153730 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-825qc"] Sep 10 15:32:38.175520 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.175495 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-825qc"] Sep 10 15:32:38.175657 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.175591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.178454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.178434 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 15:32:38.179231 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.179210 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vwfpc\"" Sep 10 15:32:38.179341 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.179238 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 15:32:38.262608 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.262578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zq5\" (UniqueName: \"kubernetes.io/projected/d1fc92b1-c638-49f5-a73f-6a5fcc6e8960-kube-api-access-g2zq5\") pod \"cert-manager-webhook-6486ffd997-825qc\" (UID: \"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960\") " pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.262761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.262656 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1fc92b1-c638-49f5-a73f-6a5fcc6e8960-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-825qc\" (UID: \"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960\") " pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.277822 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.277793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" event={"ID":"85de4b73-b795-455d-835c-17ad8fcaea3d","Type":"ContainerStarted","Data":"76590fb3286546c68d02b1d87e5f71dffa9aeeee2dd1005ff1950e75f9adda1d"} Sep 10 15:32:38.301765 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.301718 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t689z" podStartSLOduration=2.292338405 podStartE2EDuration="4.30170188s" totalImagesPullingTime="2.009363475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:38.300284115 +0000 UTC m=+328.083497180" watchObservedRunningTime="2026-09-10 15:32:38.30170188 +0000 UTC m=+328.084914944" Sep 10 15:32:38.363323 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.363288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1fc92b1-c638-49f5-a73f-6a5fcc6e8960-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-825qc\" (UID: \"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960\") " pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.363605 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.363576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zq5\" (UniqueName: \"kubernetes.io/projected/d1fc92b1-c638-49f5-a73f-6a5fcc6e8960-kube-api-access-g2zq5\") pod \"cert-manager-webhook-6486ffd997-825qc\" (UID: \"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960\") " pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.373212 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.373191 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1fc92b1-c638-49f5-a73f-6a5fcc6e8960-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-825qc\" (UID: \"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960\") " pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.373506 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.373485 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zq5\" (UniqueName: \"kubernetes.io/projected/d1fc92b1-c638-49f5-a73f-6a5fcc6e8960-kube-api-access-g2zq5\") pod \"cert-manager-webhook-6486ffd997-825qc\" (UID: \"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960\") " pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.494831 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.494757 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:38.793093 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:38.793020 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-825qc"] Sep 10 15:32:38.795805 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:32:38.795772 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1fc92b1_c638_49f5_a73f_6a5fcc6e8960.slice/crio-0076d5c07a54a3728a27a090d0ceb79bf55a5d34ff8fc9a87acb345dad36f33c WatchSource:0}: Error finding container 0076d5c07a54a3728a27a090d0ceb79bf55a5d34ff8fc9a87acb345dad36f33c: Status 404 returned error can't find the container with id 0076d5c07a54a3728a27a090d0ceb79bf55a5d34ff8fc9a87acb345dad36f33c Sep 10 15:32:39.289327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:39.289292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" event={"ID":"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960","Type":"ContainerStarted","Data":"0076d5c07a54a3728a27a090d0ceb79bf55a5d34ff8fc9a87acb345dad36f33c"} Sep 10 15:32:42.306386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:42.306348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" event={"ID":"d1fc92b1-c638-49f5-a73f-6a5fcc6e8960","Type":"ContainerStarted","Data":"3aa055eacc5665ca0b4772abdcef30cd096bfba7c60ad656b15f09edef7d6211"} Sep 10 15:32:42.306832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:42.306476 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:42.327828 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:42.327782 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" podStartSLOduration=1.625931408 podStartE2EDuration="4.327770758s" totalImagesPullingTime="2.70183935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:42.326303151 +0000 UTC m=+332.109516214" watchObservedRunningTime="2026-09-10 15:32:42.327770758 +0000 UTC m=+332.110983823" Sep 10 15:32:48.313502 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:48.313466 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-825qc" Sep 10 15:32:49.158087 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.158055 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xst26"] Sep 10 15:32:49.160412 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.160396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.162966 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.162947 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hpzs5\"" Sep 10 15:32:49.169592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.169571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xst26"] Sep 10 15:32:49.257988 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.257953 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh97l\" (UniqueName: \"kubernetes.io/projected/beb79a4d-a7e1-49b8-9ad8-a0c6d530777b-kube-api-access-hh97l\") pod \"cert-manager-55b4d954d5-xst26\" (UID: \"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b\") " pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.258256 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.258111 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/beb79a4d-a7e1-49b8-9ad8-a0c6d530777b-bound-sa-token\") pod \"cert-manager-55b4d954d5-xst26\" (UID: \"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b\") " pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.359300 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.359261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/beb79a4d-a7e1-49b8-9ad8-a0c6d530777b-bound-sa-token\") pod \"cert-manager-55b4d954d5-xst26\" (UID: \"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b\") " pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.359300 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.359302 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh97l\" (UniqueName: \"kubernetes.io/projected/beb79a4d-a7e1-49b8-9ad8-a0c6d530777b-kube-api-access-hh97l\") pod \"cert-manager-55b4d954d5-xst26\" (UID: \"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b\") " pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.368159 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.368135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh97l\" (UniqueName: \"kubernetes.io/projected/beb79a4d-a7e1-49b8-9ad8-a0c6d530777b-kube-api-access-hh97l\") pod \"cert-manager-55b4d954d5-xst26\" (UID: \"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b\") " pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.368281 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.368263 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/beb79a4d-a7e1-49b8-9ad8-a0c6d530777b-bound-sa-token\") pod \"cert-manager-55b4d954d5-xst26\" (UID: \"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b\") " pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.469595 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.469514 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xst26" Sep 10 15:32:49.738867 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:49.738796 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xst26"] Sep 10 15:32:49.744366 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:32:49.744329 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeb79a4d_a7e1_49b8_9ad8_a0c6d530777b.slice/crio-60c187e36d2cb4f1fedb20f116cde0ed85fa5692ec600502ffd1bc046c11202a WatchSource:0}: Error finding container 60c187e36d2cb4f1fedb20f116cde0ed85fa5692ec600502ffd1bc046c11202a: Status 404 returned error can't find the container with id 60c187e36d2cb4f1fedb20f116cde0ed85fa5692ec600502ffd1bc046c11202a Sep 10 15:32:50.343299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:50.343257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xst26" event={"ID":"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b","Type":"ContainerStarted","Data":"8bdb46929b932ed45fb3af1423854b7aea1ef235a7aab83b5c9d119667388e97"} Sep 10 15:32:50.343299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:50.343307 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xst26" event={"ID":"beb79a4d-a7e1-49b8-9ad8-a0c6d530777b","Type":"ContainerStarted","Data":"60c187e36d2cb4f1fedb20f116cde0ed85fa5692ec600502ffd1bc046c11202a"} Sep 10 15:32:50.361240 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:32:50.361194 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xst26" podStartSLOduration=1.3611797110000001 podStartE2EDuration="1.361179711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:50.360870025 +0000 UTC m=+340.144083091" watchObservedRunningTime="2026-09-10 15:32:50.361179711 +0000 UTC m=+340.144392776" Sep 10 15:33:09.372061 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.372025 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt"] Sep 10 15:33:09.374820 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.374802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.377222 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.377197 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:33:09.378112 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.378098 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-984bz\"" Sep 10 15:33:09.378170 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.378120 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:33:09.383250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.383226 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt"] Sep 10 15:33:09.418096 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.418066 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.418236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.418109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9b64\" (UniqueName: \"kubernetes.io/projected/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-kube-api-access-s9b64\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.418236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.418170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.519004 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.518969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.519208 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.519072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.519208 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.519110 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9b64\" (UniqueName: \"kubernetes.io/projected/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-kube-api-access-s9b64\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.519431 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.519407 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.519502 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.519464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.529419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.529391 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9b64\" (UniqueName: \"kubernetes.io/projected/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-kube-api-access-s9b64\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.684350 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.684263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:09.957568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:09.957541 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt"] Sep 10 15:33:09.959663 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:33:09.959631 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d42724_c9ce_4e7e_9c95_0fd5a7f89c7b.slice/crio-d8abd19a96bf515d12e43d64d4b803c6795288d0ec289869dbb43eb09008e027 WatchSource:0}: Error finding container d8abd19a96bf515d12e43d64d4b803c6795288d0ec289869dbb43eb09008e027: Status 404 returned error can't find the container with id d8abd19a96bf515d12e43d64d4b803c6795288d0ec289869dbb43eb09008e027 Sep 10 15:33:10.430377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:10.430342 2581 generic.go:356] "Generic (PLEG): container finished" podID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerID="3d8ba34dd81eab81eb828fc83e56eb6d867d21f0323b7b37118c7b0a3efa52d1" exitCode=0 Sep 10 15:33:10.430770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:10.430427 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" event={"ID":"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b","Type":"ContainerDied","Data":"3d8ba34dd81eab81eb828fc83e56eb6d867d21f0323b7b37118c7b0a3efa52d1"} Sep 10 15:33:10.430770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:10.430482 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" event={"ID":"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b","Type":"ContainerStarted","Data":"d8abd19a96bf515d12e43d64d4b803c6795288d0ec289869dbb43eb09008e027"} Sep 10 15:33:11.436653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:11.436616 2581 generic.go:356] "Generic (PLEG): container finished" podID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerID="fb91ded8b7699df46e1b2312832983e3279e406ac4743d4773ebfceb7ad8f5d1" exitCode=0 Sep 10 15:33:11.437067 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:11.436659 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" event={"ID":"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b","Type":"ContainerDied","Data":"fb91ded8b7699df46e1b2312832983e3279e406ac4743d4773ebfceb7ad8f5d1"} Sep 10 15:33:12.451347 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:12.451308 2581 generic.go:356] "Generic (PLEG): container finished" podID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerID="8e3637abfe06dafc0ae8d351e4a898476ef051bf1a59fd0548527adca64084fb" exitCode=0 Sep 10 15:33:12.451708 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:12.451388 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" event={"ID":"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b","Type":"ContainerDied","Data":"8e3637abfe06dafc0ae8d351e4a898476ef051bf1a59fd0548527adca64084fb"} Sep 10 15:33:13.578489 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.578465 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:13.649832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.649799 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9b64\" (UniqueName: \"kubernetes.io/projected/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-kube-api-access-s9b64\") pod \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " Sep 10 15:33:13.649968 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.649899 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-bundle\") pod \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " Sep 10 15:33:13.649968 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.649916 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-util\") pod \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\" (UID: \"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b\") " Sep 10 15:33:13.650575 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.650541 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-bundle" (OuterVolumeSpecName: "bundle") pod "46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" (UID: "46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:33:13.651975 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.651956 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-kube-api-access-s9b64" (OuterVolumeSpecName: "kube-api-access-s9b64") pod "46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" (UID: "46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b"). InnerVolumeSpecName "kube-api-access-s9b64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:33:13.655400 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.655377 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-util" (OuterVolumeSpecName: "util") pod "46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" (UID: "46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:33:13.750496 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.750399 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9b64\" (UniqueName: \"kubernetes.io/projected/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-kube-api-access-s9b64\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:33:13.750496 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.750444 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-bundle\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:33:13.750496 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:13.750454 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b-util\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:33:14.464035 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:14.463987 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" event={"ID":"46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b","Type":"ContainerDied","Data":"d8abd19a96bf515d12e43d64d4b803c6795288d0ec289869dbb43eb09008e027"} Sep 10 15:33:14.464206 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:14.464049 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8abd19a96bf515d12e43d64d4b803c6795288d0ec289869dbb43eb09008e027" Sep 10 15:33:14.464206 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:14.464015 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ntmt" Sep 10 15:33:16.507662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507628 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl"] Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507945 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="extract" Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507956 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="extract" Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507968 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="util" Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507973 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="util" Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507980 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="pull" Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.507986 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="pull" Sep 10 15:33:16.508140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.508063 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d42724-c9ce-4e7e-9c95-0fd5a7f89c7b" containerName="extract" Sep 10 15:33:16.510250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.510235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.512939 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.512912 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 15:33:16.514168 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.514150 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 15:33:16.514278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.514209 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-knkfv\"" Sep 10 15:33:16.514278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.514226 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:33:16.514374 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.514323 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 15:33:16.514374 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.514338 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 15:33:16.531178 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.531153 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl"] Sep 10 15:33:16.576492 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.576456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bdf80e8-2b1a-4b50-825c-26537c768f3d-manager-config\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.576492 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.576492 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jbq\" (UniqueName: \"kubernetes.io/projected/1bdf80e8-2b1a-4b50-825c-26537c768f3d-kube-api-access-k4jbq\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.576699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.576521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bdf80e8-2b1a-4b50-825c-26537c768f3d-metrics-cert\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.576699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.576591 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bdf80e8-2b1a-4b50-825c-26537c768f3d-cert\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.677801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.677767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bdf80e8-2b1a-4b50-825c-26537c768f3d-cert\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.677968 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.677846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bdf80e8-2b1a-4b50-825c-26537c768f3d-manager-config\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.677968 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.677867 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jbq\" (UniqueName: \"kubernetes.io/projected/1bdf80e8-2b1a-4b50-825c-26537c768f3d-kube-api-access-k4jbq\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.677968 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.677889 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bdf80e8-2b1a-4b50-825c-26537c768f3d-metrics-cert\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.678547 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.678523 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1bdf80e8-2b1a-4b50-825c-26537c768f3d-manager-config\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.680321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.680300 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bdf80e8-2b1a-4b50-825c-26537c768f3d-metrics-cert\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.680321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.680318 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bdf80e8-2b1a-4b50-825c-26537c768f3d-cert\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.688250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.688225 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jbq\" (UniqueName: \"kubernetes.io/projected/1bdf80e8-2b1a-4b50-825c-26537c768f3d-kube-api-access-k4jbq\") pod \"lws-controller-manager-d87857f9b-xh6rl\" (UID: \"1bdf80e8-2b1a-4b50-825c-26537c768f3d\") " pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:16.820136 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:16.820108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:17.105131 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:17.105095 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl"] Sep 10 15:33:17.107447 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:33:17.107421 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdf80e8_2b1a_4b50_825c_26537c768f3d.slice/crio-f02de5e98faba276f461a78803763eb8c22d0d149a49efd017bf02c0d2e45ee7 WatchSource:0}: Error finding container f02de5e98faba276f461a78803763eb8c22d0d149a49efd017bf02c0d2e45ee7: Status 404 returned error can't find the container with id f02de5e98faba276f461a78803763eb8c22d0d149a49efd017bf02c0d2e45ee7 Sep 10 15:33:17.478043 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:17.477941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" event={"ID":"1bdf80e8-2b1a-4b50-825c-26537c768f3d","Type":"ContainerStarted","Data":"f02de5e98faba276f461a78803763eb8c22d0d149a49efd017bf02c0d2e45ee7"} Sep 10 15:33:19.488648 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:19.488615 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" event={"ID":"1bdf80e8-2b1a-4b50-825c-26537c768f3d","Type":"ContainerStarted","Data":"c9c25579f170aace696d315464c54f7c438346cbdaae9cb016fb1b7e7f9151b3"} Sep 10 15:33:19.489103 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:19.488731 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:19.511184 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:19.511133 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" podStartSLOduration=1.230118943 podStartE2EDuration="3.511118537s" totalImagesPullingTime="2.280999594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:19.509746683 +0000 UTC m=+369.292959976" watchObservedRunningTime="2026-09-10 15:33:19.511118537 +0000 UTC m=+369.294331601" Sep 10 15:33:21.699519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.699482 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9"] Sep 10 15:33:21.702593 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.702565 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.705297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.705272 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 15:33:21.705443 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.705327 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 15:33:21.705443 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.705331 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 15:33:21.705443 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.705397 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f7nx8\"" Sep 10 15:33:21.705443 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.705436 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 15:33:21.721367 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.721341 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9"] Sep 10 15:33:21.822158 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.822124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b8f2b89-20ee-4b22-ba54-bc183318dded-webhook-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.822385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.822185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b8f2b89-20ee-4b22-ba54-bc183318dded-manifests\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.822385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.822231 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lfd\" (UniqueName: \"kubernetes.io/projected/0b8f2b89-20ee-4b22-ba54-bc183318dded-kube-api-access-26lfd\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.822385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.822294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b8f2b89-20ee-4b22-ba54-bc183318dded-tmp\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.822385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.822321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b8f2b89-20ee-4b22-ba54-bc183318dded-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.923762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.923724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b8f2b89-20ee-4b22-ba54-bc183318dded-tmp\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.923762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.923766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b8f2b89-20ee-4b22-ba54-bc183318dded-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.924046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.923820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b8f2b89-20ee-4b22-ba54-bc183318dded-webhook-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.924046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.923870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b8f2b89-20ee-4b22-ba54-bc183318dded-manifests\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.924046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.923886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lfd\" (UniqueName: \"kubernetes.io/projected/0b8f2b89-20ee-4b22-ba54-bc183318dded-kube-api-access-26lfd\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.924317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.924288 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b8f2b89-20ee-4b22-ba54-bc183318dded-manifests\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.926104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.926078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b8f2b89-20ee-4b22-ba54-bc183318dded-tmp\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.926368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.926350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b8f2b89-20ee-4b22-ba54-bc183318dded-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.926411 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.926394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b8f2b89-20ee-4b22-ba54-bc183318dded-webhook-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:21.943485 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:21.943447 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lfd\" (UniqueName: \"kubernetes.io/projected/0b8f2b89-20ee-4b22-ba54-bc183318dded-kube-api-access-26lfd\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-zrgn9\" (UID: \"0b8f2b89-20ee-4b22-ba54-bc183318dded\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:22.013292 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:22.013206 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:22.291610 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:22.291535 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9"] Sep 10 15:33:22.295015 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:33:22.294981 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8f2b89_20ee_4b22_ba54_bc183318dded.slice/crio-bb90e202ce78081929e959fa43a7f73fe4b091d9936eecdc933a92a89c6787f5 WatchSource:0}: Error finding container bb90e202ce78081929e959fa43a7f73fe4b091d9936eecdc933a92a89c6787f5: Status 404 returned error can't find the container with id bb90e202ce78081929e959fa43a7f73fe4b091d9936eecdc933a92a89c6787f5 Sep 10 15:33:22.505792 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:22.505755 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" event={"ID":"0b8f2b89-20ee-4b22-ba54-bc183318dded","Type":"ContainerStarted","Data":"bb90e202ce78081929e959fa43a7f73fe4b091d9936eecdc933a92a89c6787f5"} Sep 10 15:33:25.524749 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:25.524712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" event={"ID":"0b8f2b89-20ee-4b22-ba54-bc183318dded","Type":"ContainerStarted","Data":"6ef27c39341cc5d945a94d3b0471838f33c6527c5ecfb56f88a7ece0cf1d6e28"} Sep 10 15:33:26.531240 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:26.531208 2581 generic.go:356] "Generic (PLEG): container finished" podID="0b8f2b89-20ee-4b22-ba54-bc183318dded" containerID="6ef27c39341cc5d945a94d3b0471838f33c6527c5ecfb56f88a7ece0cf1d6e28" exitCode=0 Sep 10 15:33:26.531586 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:26.531257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" event={"ID":"0b8f2b89-20ee-4b22-ba54-bc183318dded","Type":"ContainerDied","Data":"6ef27c39341cc5d945a94d3b0471838f33c6527c5ecfb56f88a7ece0cf1d6e28"} Sep 10 15:33:27.537973 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:27.537935 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" event={"ID":"0b8f2b89-20ee-4b22-ba54-bc183318dded","Type":"ContainerStarted","Data":"78e231a58b8c8f2b23722021078bddac124a8df7f5a19d8c771690ee8ed1ef0c"} Sep 10 15:33:27.538637 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:27.538039 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:27.565202 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:27.565153 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" podStartSLOduration=2.000469782 podStartE2EDuration="6.565137779s" totalImagesPullingTime="3.54443736s" totalInitContainerRuntime="1.020230637s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:27.563457104 +0000 UTC m=+377.346670167" watchObservedRunningTime="2026-09-10 15:33:27.565137779 +0000 UTC m=+377.348350843" Sep 10 15:33:30.496472 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:30.496441 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d87857f9b-xh6rl" Sep 10 15:33:38.544448 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:38.544410 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-zrgn9" Sep 10 15:33:49.703207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.703167 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q"] Sep 10 15:33:49.711180 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.711153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.716416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.716383 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-w6pdq\"" Sep 10 15:33:49.716551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.716394 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 15:33:49.746088 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.746065 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q"] Sep 10 15:33:49.877846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.877806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.877846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.877844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878128 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.877874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311509f8-17b2-4e64-ac62-b15e0ba9f044-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878128 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.878072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7tq\" (UniqueName: \"kubernetes.io/projected/311509f8-17b2-4e64-ac62-b15e0ba9f044-kube-api-access-xv7tq\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878234 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.878143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878334 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.878310 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.878353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878405 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.878386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.878503 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.878444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979652 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979565 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311509f8-17b2-4e64-ac62-b15e0ba9f044-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979652 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7tq\" (UniqueName: \"kubernetes.io/projected/311509f8-17b2-4e64-ac62-b15e0ba9f044-kube-api-access-xv7tq\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979652 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.979931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.979807 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980162 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980162 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980162 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980143 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980300 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980222 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980300 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980285 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980344 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.980377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.980355 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311509f8-17b2-4e64-ac62-b15e0ba9f044-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.982125 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.982100 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:49.982334 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:49.982318 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:50.002095 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:50.002065 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311509f8-17b2-4e64-ac62-b15e0ba9f044-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:50.002264 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:50.002246 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7tq\" (UniqueName: \"kubernetes.io/projected/311509f8-17b2-4e64-ac62-b15e0ba9f044-kube-api-access-xv7tq\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cznh2q\" (UID: \"311509f8-17b2-4e64-ac62-b15e0ba9f044\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:50.020981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:50.020948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:50.298199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:50.298142 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q"] Sep 10 15:33:50.300551 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:33:50.300515 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311509f8_17b2_4e64_ac62_b15e0ba9f044.slice/crio-485eaa40583a4c87c1d85042332f94385be74743308231da0ff3421a5eb4ffae WatchSource:0}: Error finding container 485eaa40583a4c87c1d85042332f94385be74743308231da0ff3421a5eb4ffae: Status 404 returned error can't find the container with id 485eaa40583a4c87c1d85042332f94385be74743308231da0ff3421a5eb4ffae Sep 10 15:33:50.654503 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:50.654420 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" event={"ID":"311509f8-17b2-4e64-ac62-b15e0ba9f044","Type":"ContainerStarted","Data":"485eaa40583a4c87c1d85042332f94385be74743308231da0ff3421a5eb4ffae"} Sep 10 15:33:52.348348 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.348311 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qmzkc"] Sep 10 15:33:52.353181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.353157 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:52.355686 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.355646 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 15:33:52.355814 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.355707 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gdvrb\"" Sep 10 15:33:52.360621 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.360584 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qmzkc"] Sep 10 15:33:52.505071 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.505035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwss2\" (UniqueName: \"kubernetes.io/projected/63ce392e-abb9-43c5-a0a9-572456e49780-kube-api-access-nwss2\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:52.505235 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.505140 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ce392e-abb9-43c5-a0a9-572456e49780-cert\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:52.606531 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.606454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwss2\" (UniqueName: \"kubernetes.io/projected/63ce392e-abb9-43c5-a0a9-572456e49780-kube-api-access-nwss2\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:52.606709 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.606532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ce392e-abb9-43c5-a0a9-572456e49780-cert\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:52.606709 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:33:52.606631 2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 15:33:52.606709 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:33:52.606701 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ce392e-abb9-43c5-a0a9-572456e49780-cert podName:63ce392e-abb9-43c5-a0a9-572456e49780 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:53.106677493 +0000 UTC m=+402.889890544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/63ce392e-abb9-43c5-a0a9-572456e49780-cert") pod "odh-model-controller-7998c67dc4-qmzkc" (UID: "63ce392e-abb9-43c5-a0a9-572456e49780") : secret "odh-model-controller-webhook-cert" not found Sep 10 15:33:52.623529 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:52.623503 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwss2\" (UniqueName: \"kubernetes.io/projected/63ce392e-abb9-43c5-a0a9-572456e49780-kube-api-access-nwss2\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:53.110966 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.110924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ce392e-abb9-43c5-a0a9-572456e49780-cert\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:53.113551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.113526 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63ce392e-abb9-43c5-a0a9-572456e49780-cert\") pod \"odh-model-controller-7998c67dc4-qmzkc\" (UID: \"63ce392e-abb9-43c5-a0a9-572456e49780\") " pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:53.180627 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.180592 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:33:53.180696 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.180664 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:33:53.180740 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.180693 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:33:53.267540 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.267511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:53.550290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.550265 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qmzkc"] Sep 10 15:33:53.552851 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:33:53.552824 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ce392e_abb9_43c5_a0a9_572456e49780.slice/crio-d53922b5c207cec5fc61030ba9e200064bf0ce4dbce22be2543708f6b8599821 WatchSource:0}: Error finding container d53922b5c207cec5fc61030ba9e200064bf0ce4dbce22be2543708f6b8599821: Status 404 returned error can't find the container with id d53922b5c207cec5fc61030ba9e200064bf0ce4dbce22be2543708f6b8599821 Sep 10 15:33:53.673258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.673223 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" event={"ID":"311509f8-17b2-4e64-ac62-b15e0ba9f044","Type":"ContainerStarted","Data":"f8c098493f302375505bedd94f419143e4a72b7ef081abd921dd9d67ea9ef57e"} Sep 10 15:33:53.674522 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.674495 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" event={"ID":"63ce392e-abb9-43c5-a0a9-572456e49780","Type":"ContainerStarted","Data":"d53922b5c207cec5fc61030ba9e200064bf0ce4dbce22be2543708f6b8599821"} Sep 10 15:33:53.700643 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:53.700594 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" podStartSLOduration=1.8227224579999999 podStartE2EDuration="4.700579686s" totalImagesPullingTime="2.877857228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:53.698862555 +0000 UTC m=+403.482075634" watchObservedRunningTime="2026-09-10 15:33:53.700579686 +0000 UTC m=+403.483792749" Sep 10 15:33:54.022144 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:54.022053 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:54.026665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:54.026640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:54.681316 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:54.681283 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:54.682698 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:54.682565 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cznh2q" Sep 10 15:33:56.693299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:56.693201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" event={"ID":"63ce392e-abb9-43c5-a0a9-572456e49780","Type":"ContainerStarted","Data":"cc83ada0914e27773e4eb99846f34e18bdf14175786756b82f94f83885c06dbb"} Sep 10 15:33:56.693802 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:56.693466 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:33:56.715419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:56.715371 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" podStartSLOduration=1.968384605 podStartE2EDuration="4.715356671s" totalImagesPullingTime="2.746972066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:56.713269256 +0000 UTC m=+406.496482321" watchObservedRunningTime="2026-09-10 15:33:56.715356671 +0000 UTC m=+406.498569734" Sep 10 15:33:59.075953 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.075914 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv"] Sep 10 15:33:59.079773 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.079753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.083650 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.083628 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4m6v8\"" Sep 10 15:33:59.083882 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.083867 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 15:33:59.084183 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.084162 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 15:33:59.084390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.084376 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 15:33:59.106611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.106581 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv"] Sep 10 15:33:59.158401 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.158401 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ms9f\" (UniqueName: \"kubernetes.io/projected/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-kube-api-access-5ms9f\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.158600 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158482 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.158600 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158499 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.158600 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158520 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.158600 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158542 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.158600 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.158576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.259490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.259369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ms9f\" (UniqueName: \"kubernetes.io/projected/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-kube-api-access-5ms9f\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.260214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.260183 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.261868 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.261833 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.261955 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.261869 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.261955 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.261837 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.261955 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.261913 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.268701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.268673 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ms9f\" (UniqueName: \"kubernetes.io/projected/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-kube-api-access-5ms9f\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.269054 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.269036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e76894-3dc5-4ad1-bfbd-be7798b903c9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xzqhv\" (UID: \"e1e76894-3dc5-4ad1-bfbd-be7798b903c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.389387 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.389304 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:33:59.712424 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:33:59.712381 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e76894_3dc5_4ad1_bfbd_be7798b903c9.slice/crio-8436774be27ab7900d4a8076a4361054d684e64a8633ba7c742dee06f14d632d WatchSource:0}: Error finding container 8436774be27ab7900d4a8076a4361054d684e64a8633ba7c742dee06f14d632d: Status 404 returned error can't find the container with id 8436774be27ab7900d4a8076a4361054d684e64a8633ba7c742dee06f14d632d Sep 10 15:33:59.713756 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:33:59.713731 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv"] Sep 10 15:34:00.713316 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:00.713276 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" event={"ID":"e1e76894-3dc5-4ad1-bfbd-be7798b903c9","Type":"ContainerStarted","Data":"8436774be27ab7900d4a8076a4361054d684e64a8633ba7c742dee06f14d632d"} Sep 10 15:34:02.305677 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:02.305642 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:34:02.305948 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:02.305716 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:34:02.726767 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:02.726642 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" event={"ID":"e1e76894-3dc5-4ad1-bfbd-be7798b903c9","Type":"ContainerStarted","Data":"adabd7b2320c3424c871ffa4760cf3d2fd5101ea01ffa901edff51b941aaa86a"} Sep 10 15:34:02.728026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:02.727986 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:34:02.752283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:02.752227 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" podStartSLOduration=1.161365671 podStartE2EDuration="3.752205718s" totalImagesPullingTime="2.590840047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:02.751288389 +0000 UTC m=+412.534501465" watchObservedRunningTime="2026-09-10 15:34:02.752205718 +0000 UTC m=+412.535418784" Sep 10 15:34:03.732931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:03.732899 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xzqhv" Sep 10 15:34:07.700954 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:34:07.700920 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qmzkc" Sep 10 15:36:03.865871 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.865797 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz"] Sep 10 15:36:03.868470 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.868453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.873077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.873050 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gnd54\"" Sep 10 15:36:03.888168 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.888139 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz"] Sep 10 15:36:03.899304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899280 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4k2\" (UniqueName: \"kubernetes.io/projected/42c88c16-0207-4f43-a013-4acfc497f484-kube-api-access-bw4k2\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899390 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899507 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899431 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899507 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899448 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899507 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c88c16-0207-4f43-a013-4acfc497f484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c88c16-0207-4f43-a013-4acfc497f484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:03.899601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:03.899585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c88c16-0207-4f43-a013-4acfc497f484-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000072 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000072 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4k2\" (UniqueName: \"kubernetes.io/projected/42c88c16-0207-4f43-a013-4acfc497f484-kube-api-access-bw4k2\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000307 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000307 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000307 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000307 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c88c16-0207-4f43-a013-4acfc497f484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c88c16-0207-4f43-a013-4acfc497f484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c88c16-0207-4f43-a013-4acfc497f484-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000525 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000521 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.000947 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000925 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.001106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.000987 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.001205 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.001185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42c88c16-0207-4f43-a013-4acfc497f484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.002671 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.002647 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42c88c16-0207-4f43-a013-4acfc497f484-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.003036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.002986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42c88c16-0207-4f43-a013-4acfc497f484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.012373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.012340 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42c88c16-0207-4f43-a013-4acfc497f484-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.013186 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.013161 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4k2\" (UniqueName: \"kubernetes.io/projected/42c88c16-0207-4f43-a013-4acfc497f484-kube-api-access-bw4k2\") pod \"maas-default-gateway-openshift-default-5ccb968c66-md7kz\" (UID: \"42c88c16-0207-4f43-a013-4acfc497f484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.178908 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.178831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:04.459875 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.459847 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz"] Sep 10 15:36:04.461323 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:36:04.461289 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c88c16_0207_4f43_a013_4acfc497f484.slice/crio-c0acf30bea0e489a2960e877d65d17007c7ad8b2e85318792526788377acb153 WatchSource:0}: Error finding container c0acf30bea0e489a2960e877d65d17007c7ad8b2e85318792526788377acb153: Status 404 returned error can't find the container with id c0acf30bea0e489a2960e877d65d17007c7ad8b2e85318792526788377acb153 Sep 10 15:36:04.465810 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.465782 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:36:04.465884 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.465849 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:36:04.465884 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:04.465880 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:36:05.328616 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:05.328578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" event={"ID":"42c88c16-0207-4f43-a013-4acfc497f484","Type":"ContainerStarted","Data":"435ac62a19912a5813807b98179d926cc0913b8b277ede6f5ee4653cf9f3e280"} Sep 10 15:36:05.329018 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:05.328651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" event={"ID":"42c88c16-0207-4f43-a013-4acfc497f484","Type":"ContainerStarted","Data":"c0acf30bea0e489a2960e877d65d17007c7ad8b2e85318792526788377acb153"} Sep 10 15:36:05.380684 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:05.380616 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" podStartSLOduration=2.380600822 podStartE2EDuration="2.380600822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:36:05.376666268 +0000 UTC m=+535.159879353" watchObservedRunningTime="2026-09-10 15:36:05.380600822 +0000 UTC m=+535.163813885" Sep 10 15:36:06.179885 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:06.179849 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:06.184376 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:06.184347 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:06.333747 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:06.333712 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:36:06.334678 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:36:06.334659 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:37:20.879379 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:20.879341 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-shkpf"] Sep 10 15:37:20.881796 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:20.881777 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:20.884236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:20.884214 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 15:37:20.884236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:20.884230 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9gcvc\"" Sep 10 15:37:20.893761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:20.893734 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-shkpf"] Sep 10 15:37:21.005648 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.005612 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e23a13-c442-42e2-880f-9de40551873b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-shkpf\" (UID: \"e1e23a13-c442-42e2-880f-9de40551873b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.005843 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.005729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7n7\" (UniqueName: \"kubernetes.io/projected/e1e23a13-c442-42e2-880f-9de40551873b-kube-api-access-9p7n7\") pod \"payload-processing-5b54b7fbb7-shkpf\" (UID: \"e1e23a13-c442-42e2-880f-9de40551873b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.106310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.106274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e23a13-c442-42e2-880f-9de40551873b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-shkpf\" (UID: \"e1e23a13-c442-42e2-880f-9de40551873b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.106476 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.106420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7n7\" (UniqueName: \"kubernetes.io/projected/e1e23a13-c442-42e2-880f-9de40551873b-kube-api-access-9p7n7\") pod \"payload-processing-5b54b7fbb7-shkpf\" (UID: \"e1e23a13-c442-42e2-880f-9de40551873b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.106946 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.106926 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e23a13-c442-42e2-880f-9de40551873b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-shkpf\" (UID: \"e1e23a13-c442-42e2-880f-9de40551873b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.125728 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.125693 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7n7\" (UniqueName: \"kubernetes.io/projected/e1e23a13-c442-42e2-880f-9de40551873b-kube-api-access-9p7n7\") pod \"payload-processing-5b54b7fbb7-shkpf\" (UID: \"e1e23a13-c442-42e2-880f-9de40551873b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.192155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.192077 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:21.468634 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.468596 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-shkpf"] Sep 10 15:37:21.470981 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:37:21.470951 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e23a13_c442_42e2_880f_9de40551873b.slice/crio-b5347ddaa1dbb40d288eff6a6fc48145d6f9d7ed8d3794932f1e94db443bb463 WatchSource:0}: Error finding container b5347ddaa1dbb40d288eff6a6fc48145d6f9d7ed8d3794932f1e94db443bb463: Status 404 returned error can't find the container with id b5347ddaa1dbb40d288eff6a6fc48145d6f9d7ed8d3794932f1e94db443bb463 Sep 10 15:37:21.473019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.472987 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:37:21.691064 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:21.691030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" event={"ID":"e1e23a13-c442-42e2-880f-9de40551873b","Type":"ContainerStarted","Data":"b5347ddaa1dbb40d288eff6a6fc48145d6f9d7ed8d3794932f1e94db443bb463"} Sep 10 15:37:23.705389 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:23.705352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" event={"ID":"e1e23a13-c442-42e2-880f-9de40551873b","Type":"ContainerStarted","Data":"3a1ce56b6e29a05535da84b0ba56ca045a02f449bb6bcbc1a3240cb2277944f8"} Sep 10 15:37:23.705755 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:23.705577 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:23.725545 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:23.725490 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" podStartSLOduration=1.6847737010000001 podStartE2EDuration="3.72547337s" totalImagesPullingTime="2.040699669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:23.72470054 +0000 UTC m=+613.507913619" watchObservedRunningTime="2026-09-10 15:37:23.72547337 +0000 UTC m=+613.508686434" Sep 10 15:37:30.743007 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:30.742908 2581 generic.go:356] "Generic (PLEG): container finished" podID="42c88c16-0207-4f43-a013-4acfc497f484" containerID="435ac62a19912a5813807b98179d926cc0913b8b277ede6f5ee4653cf9f3e280" exitCode=0 Sep 10 15:37:30.743395 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:30.742987 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" event={"ID":"42c88c16-0207-4f43-a013-4acfc497f484","Type":"ContainerDied","Data":"435ac62a19912a5813807b98179d926cc0913b8b277ede6f5ee4653cf9f3e280"} Sep 10 15:37:30.743437 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:30.743398 2581 scope.go:117] "RemoveContainer" containerID="435ac62a19912a5813807b98179d926cc0913b8b277ede6f5ee4653cf9f3e280" Sep 10 15:37:30.744378 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:30.744354 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:37:30.744447 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:30.744434 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:37:30.744490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:30.744468 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:37:31.749367 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:31.749334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" event={"ID":"42c88c16-0207-4f43-a013-4acfc497f484","Type":"ContainerStarted","Data":"dfa3a2c5173844a54411ac359804e8fde5dbc8ed6970fb5da313b4ffb1626d74"} Sep 10 15:37:32.180008 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:32.179966 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:37:32.181300 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:32.181273 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-md7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 15:37:32.181400 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:32.181323 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" podUID="42c88c16-0207-4f43-a013-4acfc497f484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 15:37:33.179944 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:33.179843 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-md7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 15:37:33.180446 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:33.179933 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" podUID="42c88c16-0207-4f43-a013-4acfc497f484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 15:37:34.179786 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:34.179743 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-md7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 15:37:34.179946 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:34.179804 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" podUID="42c88c16-0207-4f43-a013-4acfc497f484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 15:37:34.713451 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:34.713420 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-shkpf" Sep 10 15:37:35.180236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:35.180197 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-md7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 15:37:35.180427 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:35.180282 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" podUID="42c88c16-0207-4f43-a013-4acfc497f484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 15:37:36.179731 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:36.179691 2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-md7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 15:37:36.180158 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:36.179748 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" podUID="42c88c16-0207-4f43-a013-4acfc497f484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 15:37:36.334501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:36.334466 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:37:37.183543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:37.183512 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:37:37.184368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:37:37.184349 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-md7kz" Sep 10 15:39:11.209683 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.209649 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d"] Sep 10 15:39:11.212271 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.212250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.214804 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.214781 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vwb6m\"" Sep 10 15:39:11.215938 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.215913 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 15:39:11.215938 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.215927 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 15:39:11.216119 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.215938 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 15:39:11.232649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.232624 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d"] Sep 10 15:39:11.323236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.323200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.323414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.323265 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be88b9-90f5-4d46-8b22-f26ca8489e31-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.323456 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.323405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlwm\" (UniqueName: \"kubernetes.io/projected/a0be88b9-90f5-4d46-8b22-f26ca8489e31-kube-api-access-rwlwm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.323456 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.323437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.323526 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.323465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.323559 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.323533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424275 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlwm\" (UniqueName: \"kubernetes.io/projected/a0be88b9-90f5-4d46-8b22-f26ca8489e31-kube-api-access-rwlwm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424285 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424537 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424537 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be88b9-90f5-4d46-8b22-f26ca8489e31-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424692 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424639 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424745 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.424907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.424875 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.426653 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.426634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be88b9-90f5-4d46-8b22-f26ca8489e31-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.427008 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.426978 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be88b9-90f5-4d46-8b22-f26ca8489e31-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.433626 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.433605 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlwm\" (UniqueName: \"kubernetes.io/projected/a0be88b9-90f5-4d46-8b22-f26ca8489e31-kube-api-access-rwlwm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-84v4d\" (UID: \"a0be88b9-90f5-4d46-8b22-f26ca8489e31\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.524118 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.524031 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:11.806372 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:11.806346 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d"] Sep 10 15:39:11.807749 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:39:11.807724 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0be88b9_90f5_4d46_8b22_f26ca8489e31.slice/crio-b23e2b96fc8a8c40513b09f617bfe019fc585bd219c8d54195e9fe01f3cee7c2 WatchSource:0}: Error finding container b23e2b96fc8a8c40513b09f617bfe019fc585bd219c8d54195e9fe01f3cee7c2: Status 404 returned error can't find the container with id b23e2b96fc8a8c40513b09f617bfe019fc585bd219c8d54195e9fe01f3cee7c2 Sep 10 15:39:12.257263 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:12.257186 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" event={"ID":"a0be88b9-90f5-4d46-8b22-f26ca8489e31","Type":"ContainerStarted","Data":"b23e2b96fc8a8c40513b09f617bfe019fc585bd219c8d54195e9fe01f3cee7c2"} Sep 10 15:39:13.263941 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:13.263901 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" event={"ID":"a0be88b9-90f5-4d46-8b22-f26ca8489e31","Type":"ContainerStarted","Data":"a512ca06e9ce0080198bb2832f9340d4836ab3c5688abde6a572472164ca13b7"} Sep 10 15:39:13.283590 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:13.283493 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" podStartSLOduration=1.071878954 podStartE2EDuration="2.283477089s" totalImagesPullingTime="1.211598135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:13.281693148 +0000 UTC m=+723.064906213" watchObservedRunningTime="2026-09-10 15:39:13.283477089 +0000 UTC m=+723.066690152" Sep 10 15:39:21.524439 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:21.524387 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:21.524921 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:21.524474 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:21.536795 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:21.536768 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:39:22.320102 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:39:22.320072 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-84v4d" Sep 10 15:41:27.781584 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.781533 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b"] Sep 10 15:41:27.785393 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.785372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.787834 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.787812 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-139155-openshift-default-dockercfg-fdbxs\"" Sep 10 15:41:27.796479 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.796458 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b"] Sep 10 15:41:27.921438 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-socket\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-certs\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921498 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-data\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-podinfo\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-token\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921804 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istiod-ca-cert\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921804 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-credential-socket\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921804 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-envoy\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:27.921902 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:27.921801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ptn\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-kube-api-access-69ptn\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.022831 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.022796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-certs\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.022831 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.022832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-data\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023071 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.022857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-podinfo\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023071 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.022902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-token\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023071 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.022976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istiod-ca-cert\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023071 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-credential-socket\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-envoy\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ptn\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-kube-api-access-69ptn\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-socket\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023215 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-certs\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023282 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-data\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023620 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023597 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-credential-socket\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023711 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023665 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-socket\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.023711 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.023701 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istiod-ca-cert\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.025296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.025273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-envoy\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.025544 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.025528 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-podinfo\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.031851 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.031787 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-token\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.031960 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.031907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ptn\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-kube-api-access-69ptn\") pod \"e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.097152 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.097118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:28.375133 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.375102 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b"] Sep 10 15:41:28.378535 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:41:28.378506 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bbfd2b_67dd_4fa6_90e1_e6d17db73236.slice/crio-8678a18a99aa32eaaf7116af2dd09fb57f24516ad7db7495d1a32c5362451c94 WatchSource:0}: Error finding container 8678a18a99aa32eaaf7116af2dd09fb57f24516ad7db7495d1a32c5362451c94: Status 404 returned error can't find the container with id 8678a18a99aa32eaaf7116af2dd09fb57f24516ad7db7495d1a32c5362451c94 Sep 10 15:41:28.380761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.380731 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:28.380835 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.380814 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:28.380873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.380842 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:28.943569 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.943531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" event={"ID":"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236","Type":"ContainerStarted","Data":"f0abe35e07a99ca5ac9c23db9a9d1b5c080a76d2ad25fd49af0f1cb3de95d401"} Sep 10 15:41:28.943925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.943592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" event={"ID":"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236","Type":"ContainerStarted","Data":"8678a18a99aa32eaaf7116af2dd09fb57f24516ad7db7495d1a32c5362451c94"} Sep 10 15:41:28.987681 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:28.987634 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podStartSLOduration=1.987618085 podStartE2EDuration="1.987618085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:28.97691251 +0000 UTC m=+858.760125574" watchObservedRunningTime="2026-09-10 15:41:28.987618085 +0000 UTC m=+858.770831149" Sep 10 15:41:29.098183 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:29.098144 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:29.103107 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:29.103079 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:29.949238 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:29.949207 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:29.950171 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:29.950150 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:32.853888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.853842 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b"] Sep 10 15:41:32.858255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.858220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:32.860967 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.860894 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-139155-dockercfg-hrtz5\"" Sep 10 15:41:32.860967 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.860949 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-139155\"" Sep 10 15:41:32.868478 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.868414 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b"] Sep 10 15:41:32.973275 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.973237 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:32.973428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:32.973411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpsz\" (UniqueName: \"kubernetes.io/projected/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-kube-api-access-jtpsz\") pod \"payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.074255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.074221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.074423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.074360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpsz\" (UniqueName: \"kubernetes.io/projected/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-kube-api-access-jtpsz\") pod \"payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.075101 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.075082 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.083273 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.083250 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpsz\" (UniqueName: \"kubernetes.io/projected/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-kube-api-access-jtpsz\") pod \"payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.175432 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.175349 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.451818 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.451772 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b"] Sep 10 15:41:33.454554 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:41:33.454527 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317eb4e4_5c0b_45cb_9397_f2d5183c1e95.slice/crio-53ce619acde395adf7dfa71c7681319e1640a51602388897eaaf57e42e6045a0 WatchSource:0}: Error finding container 53ce619acde395adf7dfa71c7681319e1640a51602388897eaaf57e42e6045a0: Status 404 returned error can't find the container with id 53ce619acde395adf7dfa71c7681319e1640a51602388897eaaf57e42e6045a0 Sep 10 15:41:33.975990 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.975957 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" event={"ID":"317eb4e4-5c0b-45cb-9397-f2d5183c1e95","Type":"ContainerStarted","Data":"fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794"} Sep 10 15:41:33.976399 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.976020 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" event={"ID":"317eb4e4-5c0b-45cb-9397-f2d5183c1e95","Type":"ContainerStarted","Data":"53ce619acde395adf7dfa71c7681319e1640a51602388897eaaf57e42e6045a0"} Sep 10 15:41:33.976399 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.976128 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:33.994127 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:33.994081 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" podStartSLOduration=1.994066659 podStartE2EDuration="1.994066659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:33.992909687 +0000 UTC m=+863.776122749" watchObservedRunningTime="2026-09-10 15:41:33.994066659 +0000 UTC m=+863.777279723" Sep 10 15:41:36.288744 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.288711 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql"] Sep 10 15:41:36.292714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.292692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.295494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.295469 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7a711aa8-openshift-default-dockercfg-xrvdx\"" Sep 10 15:41:36.306883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.306856 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql"] Sep 10 15:41:36.408088 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-certs\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-credential-socket\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwkjp\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-kube-api-access-hwkjp\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-socket\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-envoy\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408308 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-data\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-podinfo\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-token\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.408523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.408452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istiod-ca-cert\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-envoy\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-data\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-podinfo\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-token\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istiod-ca-cert\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509606 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-certs\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509606 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-credential-socket\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509606 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwkjp\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-kube-api-access-hwkjp\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509606 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-socket\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509809 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-data\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.509809 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509777 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-socket\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.510023 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.509976 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-credential-socket\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.510121 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.510051 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-certs\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.510184 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.510124 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istiod-ca-cert\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.511778 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.511751 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-envoy\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.512250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.512228 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-podinfo\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.519416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.519392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-token\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.519587 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.519564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwkjp\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-kube-api-access-hwkjp\") pod \"e2e-ait-7a711aa8-openshift-default-686c9689d-svzql\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.605944 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.605915 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:36.887020 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.886979 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql"] Sep 10 15:41:36.889133 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:41:36.889105 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb47153_3a4f_479a_b1b1_e7b322179cf5.slice/crio-2fe5115a6d1e97140e48e09c1c8eec9a1dd801c0ad97619e15926a01f2dde7ed WatchSource:0}: Error finding container 2fe5115a6d1e97140e48e09c1c8eec9a1dd801c0ad97619e15926a01f2dde7ed: Status 404 returned error can't find the container with id 2fe5115a6d1e97140e48e09c1c8eec9a1dd801c0ad97619e15926a01f2dde7ed Sep 10 15:41:36.891904 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.891871 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:36.891981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.891938 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:36.891981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.891970 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:36.997105 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.997068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" event={"ID":"8bb47153-3a4f-479a-b1b1-e7b322179cf5","Type":"ContainerStarted","Data":"04a7d20815d82e651320b6f5341b54f67d9f39af91822eb43d44cf43383443ba"} Sep 10 15:41:36.997290 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:36.997134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" event={"ID":"8bb47153-3a4f-479a-b1b1-e7b322179cf5","Type":"ContainerStarted","Data":"2fe5115a6d1e97140e48e09c1c8eec9a1dd801c0ad97619e15926a01f2dde7ed"} Sep 10 15:41:37.021041 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.020966 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podStartSLOduration=1.020949779 podStartE2EDuration="1.020949779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:37.0181036 +0000 UTC m=+866.801316663" watchObservedRunningTime="2026-09-10 15:41:37.020949779 +0000 UTC m=+866.804162843" Sep 10 15:41:37.270939 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.270864 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls"] Sep 10 15:41:37.275356 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.275326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.278069 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.278042 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-bd5806-openshift-default-dockercfg-wf6s4\"" Sep 10 15:41:37.283092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.283055 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls"] Sep 10 15:41:37.316392 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-data\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-certs\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316463 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-envoy\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-credential-socket\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-token\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9hj\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-kube-api-access-4j9hj\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istiod-ca-cert\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-socket\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.316770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.316761 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-podinfo\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418231 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-credential-socket\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418411 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418267 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-token\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418411 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9hj\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-kube-api-access-4j9hj\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418411 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istiod-ca-cert\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418596 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-socket\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418596 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-podinfo\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418596 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-data\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418750 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-certs\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418750 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-envoy\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418750 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418681 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-credential-socket\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418904 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418827 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-socket\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.418960 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418906 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-certs\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.419027 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.418979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-data\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.419089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.419066 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istiod-ca-cert\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.420926 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.420902 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-envoy\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.421182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.421165 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-podinfo\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.427242 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.427218 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-token\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.427346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.427302 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9hj\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-kube-api-access-4j9hj\") pod \"e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.588477 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.588448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:37.606973 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.606939 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:37.613076 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.613052 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:37.871846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.871821 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls"] Sep 10 15:41:37.874288 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:41:37.874262 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96f20ea_2bbd_4fd6_b09b_7c167e990314.slice/crio-4dcbfb5487394c12f0c5b9272000ed79a77faf331fa1c6317a798299d6be3048 WatchSource:0}: Error finding container 4dcbfb5487394c12f0c5b9272000ed79a77faf331fa1c6317a798299d6be3048: Status 404 returned error can't find the container with id 4dcbfb5487394c12f0c5b9272000ed79a77faf331fa1c6317a798299d6be3048 Sep 10 15:41:37.877108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.877071 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:37.877197 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.877149 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:37.877197 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:37.877186 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:38.004820 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.004768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" event={"ID":"a96f20ea-2bbd-4fd6-b09b-7c167e990314","Type":"ContainerStarted","Data":"382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1"} Sep 10 15:41:38.005047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.004839 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" event={"ID":"a96f20ea-2bbd-4fd6-b09b-7c167e990314","Type":"ContainerStarted","Data":"4dcbfb5487394c12f0c5b9272000ed79a77faf331fa1c6317a798299d6be3048"} Sep 10 15:41:38.005255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.005230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:38.006356 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.006334 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:38.029468 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.029365 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podStartSLOduration=1.029344349 podStartE2EDuration="1.029344349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:38.028471762 +0000 UTC m=+867.811684846" watchObservedRunningTime="2026-09-10 15:41:38.029344349 +0000 UTC m=+867.812557415" Sep 10 15:41:38.588642 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.588611 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:38.593490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:38.593463 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:39.010119 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:39.010025 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:39.011037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:39.011017 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:41:44.789600 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.789564 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g"] Sep 10 15:41:44.793561 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.793542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:44.795836 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.795808 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-bd5806\"" Sep 10 15:41:44.795960 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.795838 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-bd5806-dockercfg-f2q22\"" Sep 10 15:41:44.802907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.802872 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g"] Sep 10 15:41:44.908651 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.908616 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9276e3-474d-4014-96b6-0d84030543c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:44.908784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.908663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rv4\" (UniqueName: \"kubernetes.io/projected/7b9276e3-474d-4014-96b6-0d84030543c1-kube-api-access-q2rv4\") pod \"payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:44.984553 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:44.984527 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:41:45.009844 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.009806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9276e3-474d-4014-96b6-0d84030543c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:45.010060 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.009918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rv4\" (UniqueName: \"kubernetes.io/projected/7b9276e3-474d-4014-96b6-0d84030543c1-kube-api-access-q2rv4\") pod \"payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:45.010745 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.010719 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9276e3-474d-4014-96b6-0d84030543c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:45.019617 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.019589 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rv4\" (UniqueName: \"kubernetes.io/projected/7b9276e3-474d-4014-96b6-0d84030543c1-kube-api-access-q2rv4\") pod \"payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:45.107485 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.107454 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:45.394346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.394315 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g"] Sep 10 15:41:45.396115 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:41:45.396074 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9276e3_474d_4014_96b6_0d84030543c1.slice/crio-ab032f87b04d78aaabf6fcedcc96619478fd343fc21465f4e66de46d3f970757 WatchSource:0}: Error finding container ab032f87b04d78aaabf6fcedcc96619478fd343fc21465f4e66de46d3f970757: Status 404 returned error can't find the container with id ab032f87b04d78aaabf6fcedcc96619478fd343fc21465f4e66de46d3f970757 Sep 10 15:41:45.991198 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.991154 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:41:45.991669 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:45.991232 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:41:46.049933 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:46.049901 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" event={"ID":"7b9276e3-474d-4014-96b6-0d84030543c1","Type":"ContainerStarted","Data":"8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f"} Sep 10 15:41:46.050120 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:46.049949 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" event={"ID":"7b9276e3-474d-4014-96b6-0d84030543c1","Type":"ContainerStarted","Data":"ab032f87b04d78aaabf6fcedcc96619478fd343fc21465f4e66de46d3f970757"} Sep 10 15:41:46.050120 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:46.050050 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:46.071859 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:46.071800 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" podStartSLOduration=2.071783569 podStartE2EDuration="2.071783569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:46.070626305 +0000 UTC m=+875.853839371" watchObservedRunningTime="2026-09-10 15:41:46.071783569 +0000 UTC m=+875.854996670" Sep 10 15:41:48.076403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:48.076370 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerID="f0abe35e07a99ca5ac9c23db9a9d1b5c080a76d2ad25fd49af0f1cb3de95d401" exitCode=0 Sep 10 15:41:48.076403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:48.076414 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" event={"ID":"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236","Type":"ContainerDied","Data":"f0abe35e07a99ca5ac9c23db9a9d1b5c080a76d2ad25fd49af0f1cb3de95d401"} Sep 10 15:41:48.077029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:48.076805 2581 scope.go:117] "RemoveContainer" containerID="f0abe35e07a99ca5ac9c23db9a9d1b5c080a76d2ad25fd49af0f1cb3de95d401" Sep 10 15:41:48.078093 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:48.078053 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:48.078192 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:48.078148 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:48.078235 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:48.078194 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:49.088384 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:49.088345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" event={"ID":"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236","Type":"ContainerStarted","Data":"edaeb35b84da853bbba754b27f90fa5e169274b587e8529927b4a85b4d4ab410"} Sep 10 15:41:49.097813 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:49.097780 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:49.098151 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:49.098130 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:41:49.098318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:49.098295 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 15:41:50.097956 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:50.097911 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:41:50.098442 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:50.098007 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 15:41:51.098010 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:51.097955 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:41:51.098471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:51.098052 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 15:41:52.106047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:52.100472 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:41:52.106047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:52.100538 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 15:41:53.098297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:53.098263 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:41:53.098468 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:53.098318 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 15:41:54.047188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:54.047141 2581 patch_prober.go:28] interesting pod/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:41:54.047583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:54.047212 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:41:54.098026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:54.097975 2581 patch_prober.go:28] interesting pod/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:41:54.098199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:54.098053 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 15:41:55.101759 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:55.101723 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:55.102236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:55.102222 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:55.103030 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:55.102988 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:41:57.072172 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.072139 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:41:57.143907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.143874 2581 generic.go:356] "Generic (PLEG): container finished" podID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerID="04a7d20815d82e651320b6f5341b54f67d9f39af91822eb43d44cf43383443ba" exitCode=0 Sep 10 15:41:57.144088 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.143930 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" event={"ID":"8bb47153-3a4f-479a-b1b1-e7b322179cf5","Type":"ContainerDied","Data":"04a7d20815d82e651320b6f5341b54f67d9f39af91822eb43d44cf43383443ba"} Sep 10 15:41:57.144347 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.144331 2581 scope.go:117] "RemoveContainer" containerID="04a7d20815d82e651320b6f5341b54f67d9f39af91822eb43d44cf43383443ba" Sep 10 15:41:57.145443 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.145414 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:57.145514 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.145499 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:57.145553 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.145535 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:41:57.805018 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.803831 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg"] Sep 10 15:41:57.806675 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.806656 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:57.809160 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.809141 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-90ca97\"" Sep 10 15:41:57.816485 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.816455 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg"] Sep 10 15:41:57.961620 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.961582 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqxf\" (UniqueName: \"kubernetes.io/projected/e8e1a09c-0288-485f-8fd8-7df3b76367b7-kube-api-access-hdqxf\") pod \"payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:57.962789 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:57.962764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1a09c-0288-485f-8fd8-7df3b76367b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:58.065015 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.064894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1a09c-0288-485f-8fd8-7df3b76367b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:58.066383 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.066351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqxf\" (UniqueName: \"kubernetes.io/projected/e8e1a09c-0288-485f-8fd8-7df3b76367b7-kube-api-access-hdqxf\") pod \"payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:58.066982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.066142 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1a09c-0288-485f-8fd8-7df3b76367b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:58.076763 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.076735 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqxf\" (UniqueName: \"kubernetes.io/projected/e8e1a09c-0288-485f-8fd8-7df3b76367b7-kube-api-access-hdqxf\") pod \"payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:58.119706 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.119668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:58.159024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.158705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" event={"ID":"8bb47153-3a4f-479a-b1b1-e7b322179cf5","Type":"ContainerStarted","Data":"85ac6198787120368f99e82dc803422335825da29ef1b1f54e02298c3001b87c"} Sep 10 15:41:58.448359 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.447961 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg"] Sep 10 15:41:58.609016 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.606495 2581 patch_prober.go:28] interesting pod/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:41:58.609016 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.606555 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:41:58.609016 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.606574 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 10 15:41:58.776538 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.776427 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86"] Sep 10 15:41:58.780182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.780150 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:41:58.784407 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.784378 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:41:58.784647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.784627 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:41:58.794520 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.794487 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86"] Sep 10 15:41:58.818697 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.818633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86\" (UID: \"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:41:58.919500 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.919448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86\" (UID: \"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:41:58.920089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:58.920060 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86\" (UID: \"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:41:59.103891 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.103857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:41:59.212281 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.212231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" event={"ID":"e8e1a09c-0288-485f-8fd8-7df3b76367b7","Type":"ContainerStarted","Data":"571ef218c6f0e79c14a659d4d6d0e56ce7aaa379894283fcb7d7b32b2ca6e82d"} Sep 10 15:41:59.212551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.212516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" event={"ID":"e8e1a09c-0288-485f-8fd8-7df3b76367b7","Type":"ContainerStarted","Data":"1b722e66966c2980be5373851fcf8171e76dc2c02f8c873a23f56605ba200296"} Sep 10 15:41:59.212759 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.212743 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:41:59.243021 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.242608 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" podStartSLOduration=2.242586362 podStartE2EDuration="2.242586362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:59.23767351 +0000 UTC m=+889.020886576" watchObservedRunningTime="2026-09-10 15:41:59.242586362 +0000 UTC m=+889.025799427" Sep 10 15:41:59.417279 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.417253 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86"] Sep 10 15:41:59.606784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.606735 2581 patch_prober.go:28] interesting pod/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:41:59.606962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:41:59.606799 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 10 15:42:00.209523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:00.209491 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" event={"ID":"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8","Type":"ContainerStarted","Data":"667450ac343057636c830916aaa0af466c913858aa5e3ec6f92d0e2297ff254f"} Sep 10 15:42:00.606933 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:00.606894 2581 patch_prober.go:28] interesting pod/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:42:00.607125 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:00.606957 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 10 15:42:01.606902 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:01.606829 2581 patch_prober.go:28] interesting pod/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:42:01.606902 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:01.606887 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 10 15:42:02.607056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:02.607026 2581 patch_prober.go:28] interesting pod/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:42:02.607465 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:02.607078 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 10 15:42:03.235258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:03.235164 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" event={"ID":"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8","Type":"ContainerStarted","Data":"23280b6a706396b1791dbb2c992613d157515905f571b068781a69ff64671f89"} Sep 10 15:42:03.265176 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:03.265109 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" podStartSLOduration=2.09170665 podStartE2EDuration="5.265086806s" totalImagesPullingTime="3.173380156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:03.262342674 +0000 UTC m=+893.045555739" watchObservedRunningTime="2026-09-10 15:42:03.265086806 +0000 UTC m=+893.048299870" Sep 10 15:42:03.610831 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:03.610798 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:42:03.611296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:03.611198 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:42:03.612027 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:03.611987 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:42:04.243138 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:04.243097 2581 generic.go:356] "Generic (PLEG): container finished" podID="727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8" containerID="23280b6a706396b1791dbb2c992613d157515905f571b068781a69ff64671f89" exitCode=0 Sep 10 15:42:04.244378 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:04.243178 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" event={"ID":"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8","Type":"ContainerDied","Data":"23280b6a706396b1791dbb2c992613d157515905f571b068781a69ff64671f89"} Sep 10 15:42:05.250576 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.250545 2581 generic.go:356] "Generic (PLEG): container finished" podID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerID="382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1" exitCode=0 Sep 10 15:42:05.250977 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.250627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" event={"ID":"a96f20ea-2bbd-4fd6-b09b-7c167e990314","Type":"ContainerDied","Data":"382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1"} Sep 10 15:42:05.251342 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.251208 2581 scope.go:117] "RemoveContainer" containerID="382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1" Sep 10 15:42:05.252329 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.252299 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:05.252438 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.252373 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:05.252438 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.252411 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:05.405255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.405231 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:42:05.512248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.512167 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8-maas-api-service-ca\") pod \"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8\" (UID: \"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8\") " Sep 10 15:42:05.512634 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.512607 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8" (UID: "727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:42:05.612930 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:05.612896 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:06.266313 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.266277 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" Sep 10 15:42:06.266803 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.266312 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86" event={"ID":"727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8","Type":"ContainerDied","Data":"667450ac343057636c830916aaa0af466c913858aa5e3ec6f92d0e2297ff254f"} Sep 10 15:42:06.266803 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.266375 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667450ac343057636c830916aaa0af466c913858aa5e3ec6f92d0e2297ff254f" Sep 10 15:42:06.279097 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.278977 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" event={"ID":"a96f20ea-2bbd-4fd6-b09b-7c167e990314","Type":"ContainerStarted","Data":"76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967"} Sep 10 15:42:06.589578 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.589540 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:42:06.590794 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.590771 2581 patch_prober.go:28] interesting pod/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 15:42:06.590873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:06.590820 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 15:42:07.588849 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:07.588814 2581 patch_prober.go:28] interesting pod/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 15:42:07.589274 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:07.588863 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 15:42:08.589494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:08.589455 2581 patch_prober.go:28] interesting pod/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 15:42:08.589874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:08.589517 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 15:42:09.010390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:09.010299 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:42:09.589574 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:09.589539 2581 patch_prober.go:28] interesting pod/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 15:42:09.589982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:09.589606 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 15:42:09.936046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:09.935920 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86"] Sep 10 15:42:09.944331 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:09.944280 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a711aa8-jrb86"] Sep 10 15:42:10.074141 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:10.074106 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql"] Sep 10 15:42:10.074622 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:10.074591 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" containerID="cri-o://85ac6198787120368f99e82dc803422335825da29ef1b1f54e02298c3001b87c" gracePeriod=30 Sep 10 15:42:10.588899 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:10.588865 2581 patch_prober.go:28] interesting pod/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 15:42:10.589085 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:10.588933 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 15:42:10.789958 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:10.789924 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8" path="/var/lib/kubelet/pods/727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8/volumes" Sep 10 15:42:11.025147 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.025062 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4"] Sep 10 15:42:11.025970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.025944 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8" containerName="revoke-keys" Sep 10 15:42:11.025970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.025971 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8" containerName="revoke-keys" Sep 10 15:42:11.026171 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.026138 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="727dc7d3-1a4e-4e44-94f7-afe9ae3ce3c8" containerName="revoke-keys" Sep 10 15:42:11.030076 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.030056 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.032530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.032510 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f89c4f3a-gw-openshift-default-dockercfg-fvbz6\"" Sep 10 15:42:11.043815 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.043743 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4"] Sep 10 15:42:11.201609 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75607121-461e-460f-a059-4c10302a323c-istiod-ca-cert\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201780 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-socket\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201780 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75607121-461e-460f-a059-4c10302a323c-istio-podinfo\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201780 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-envoy\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201804 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-certs\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-credential-socket\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2hs\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-kube-api-access-nl2hs\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.201895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-istio-token\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.202066 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.201918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-data\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.220421 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.220389 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:42:11.302572 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-envoy\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.302760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-certs\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.302760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-credential-socket\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.302760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2hs\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-kube-api-access-nl2hs\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.302760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-istio-token\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303023 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-data\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303023 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75607121-461e-460f-a059-4c10302a323c-istiod-ca-cert\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303023 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-socket\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303023 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.302913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75607121-461e-460f-a059-4c10302a323c-istio-podinfo\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.303053 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-credential-socket\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.303080 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-certs\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303382 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.303301 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-data\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303438 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.303386 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-socket\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.303703 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.303679 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75607121-461e-460f-a059-4c10302a323c-istiod-ca-cert\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.305217 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.305186 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-envoy\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.305478 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.305462 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75607121-461e-460f-a059-4c10302a323c-istio-podinfo\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.316870 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.316844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2hs\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-kube-api-access-nl2hs\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.317802 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.317780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-istio-token\") pod \"e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.345874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.345849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:11.601562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.601469 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:42:11.603679 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.603604 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:42:11.640714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.640452 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4"] Sep 10 15:42:11.646857 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.646702 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:11.646857 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.646796 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:11.646857 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:11.646841 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:12.316788 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:12.316750 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" event={"ID":"75607121-461e-460f-a059-4c10302a323c","Type":"ContainerStarted","Data":"ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718"} Sep 10 15:42:12.317194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:12.316806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" event={"ID":"75607121-461e-460f-a059-4c10302a323c","Type":"ContainerStarted","Data":"988be577fb850b203f8d4d9bd5018bc1109acdf5e231758aef7a26066e1aa227"} Sep 10 15:42:12.338695 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:12.338647 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podStartSLOduration=1.33863278 podStartE2EDuration="1.33863278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:12.337190308 +0000 UTC m=+902.120403379" watchObservedRunningTime="2026-09-10 15:42:12.33863278 +0000 UTC m=+902.121845881" Sep 10 15:42:12.346737 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:12.346714 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:12.351761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:12.351738 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:13.323147 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:13.323118 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:13.324050 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:13.324032 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:15.336757 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.336722 2581 generic.go:356] "Generic (PLEG): container finished" podID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerID="85ac6198787120368f99e82dc803422335825da29ef1b1f54e02298c3001b87c" exitCode=0 Sep 10 15:42:15.337211 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.336802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" event={"ID":"8bb47153-3a4f-479a-b1b1-e7b322179cf5","Type":"ContainerDied","Data":"85ac6198787120368f99e82dc803422335825da29ef1b1f54e02298c3001b87c"} Sep 10 15:42:15.337211 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.336876 2581 scope.go:117] "RemoveContainer" containerID="04a7d20815d82e651320b6f5341b54f67d9f39af91822eb43d44cf43383443ba" Sep 10 15:42:15.454346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.454319 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:42:15.551096 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551071 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-token\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551105 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwkjp\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-kube-api-access-hwkjp\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551140 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-certs\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551175 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-credential-socket\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551202 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-podinfo\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551227 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-envoy\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551278 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-data\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551331 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-socket\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551371 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istiod-ca-cert\") pod \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\" (UID: \"8bb47153-3a4f-479a-b1b1-e7b322179cf5\") " Sep 10 15:42:15.551641 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551510 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:15.551641 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551524 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:15.551759 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551727 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:15.551759 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551740 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:15.551866 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551803 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.551866 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551822 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.551866 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551835 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.551866 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.551828 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:42:15.553568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.553536 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:15.553702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.553596 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:42:15.553702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.553608 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-kube-api-access-hwkjp" (OuterVolumeSpecName: "kube-api-access-hwkjp") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "kube-api-access-hwkjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:42:15.553702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.553608 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bb47153-3a4f-479a-b1b1-e7b322179cf5" (UID: "8bb47153-3a4f-479a-b1b1-e7b322179cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:42:15.652339 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.652301 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.652339 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.652333 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwkjp\" (UniqueName: \"kubernetes.io/projected/8bb47153-3a4f-479a-b1b1-e7b322179cf5-kube-api-access-hwkjp\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.652339 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.652344 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.652583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.652353 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.652583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.652361 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:15.652583 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:15.652368 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb47153-3a4f-479a-b1b1-e7b322179cf5-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:16.343730 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:16.343699 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" Sep 10 15:42:16.344167 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:16.343698 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql" event={"ID":"8bb47153-3a4f-479a-b1b1-e7b322179cf5","Type":"ContainerDied","Data":"2fe5115a6d1e97140e48e09c1c8eec9a1dd801c0ad97619e15926a01f2dde7ed"} Sep 10 15:42:16.344167 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:16.343825 2581 scope.go:117] "RemoveContainer" containerID="85ac6198787120368f99e82dc803422335825da29ef1b1f54e02298c3001b87c" Sep 10 15:42:16.367051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:16.366984 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql"] Sep 10 15:42:16.373472 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:16.373445 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7a711aa8-openshift-default-686c9689d-svzql"] Sep 10 15:42:16.788987 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:16.788912 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" path="/var/lib/kubelet/pods/8bb47153-3a4f-479a-b1b1-e7b322179cf5/volumes" Sep 10 15:42:34.446981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:34.446943 2581 generic.go:356] "Generic (PLEG): container finished" podID="75607121-461e-460f-a059-4c10302a323c" containerID="ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718" exitCode=0 Sep 10 15:42:34.447404 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:34.447019 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" event={"ID":"75607121-461e-460f-a059-4c10302a323c","Type":"ContainerDied","Data":"ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718"} Sep 10 15:42:34.447447 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:34.447436 2581 scope.go:117] "RemoveContainer" containerID="ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718" Sep 10 15:42:34.448666 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:34.448638 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:34.448728 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:34.448718 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:34.448769 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:34.448753 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:42:35.453936 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:35.453892 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" event={"ID":"75607121-461e-460f-a059-4c10302a323c","Type":"ContainerStarted","Data":"c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac"} Sep 10 15:42:36.346635 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:36.346598 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:36.346835 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:36.346812 2581 patch_prober.go:28] interesting pod/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:42:36.346887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:36.346864 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:42:37.346679 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:37.346639 2581 patch_prober.go:28] interesting pod/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:42:37.347106 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:37.346701 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:42:38.347129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.347093 2581 patch_prober.go:28] interesting pod/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:42:38.347498 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.347156 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:42:38.858895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.858837 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7"] Sep 10 15:42:38.859588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.859562 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" Sep 10 15:42:38.859588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.859588 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" Sep 10 15:42:38.859770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.859612 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" Sep 10 15:42:38.859770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.859621 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" Sep 10 15:42:38.859770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.859744 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" Sep 10 15:42:38.859770 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.859757 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb47153-3a4f-479a-b1b1-e7b322179cf5" containerName="istio-proxy" Sep 10 15:42:38.862488 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.862459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:38.867624 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.867596 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:42:38.867893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.867868 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:42:38.870877 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.870845 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7"] Sep 10 15:42:38.989518 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:38.989477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b953aef1-1fd7-4744-bca2-0bd1b234d35a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7\" (UID: \"b953aef1-1fd7-4744-bca2-0bd1b234d35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:39.090305 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.090266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b953aef1-1fd7-4744-bca2-0bd1b234d35a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7\" (UID: \"b953aef1-1fd7-4744-bca2-0bd1b234d35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:39.090932 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.090908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b953aef1-1fd7-4744-bca2-0bd1b234d35a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7\" (UID: \"b953aef1-1fd7-4744-bca2-0bd1b234d35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:39.178815 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.178724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:39.346981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.346942 2581 patch_prober.go:28] interesting pod/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:42:39.347195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.347027 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:42:39.449925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.449900 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7"] Sep 10 15:42:39.486123 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:39.486090 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" event={"ID":"b953aef1-1fd7-4744-bca2-0bd1b234d35a","Type":"ContainerStarted","Data":"945d48ed071001596a5403ac060c8b9c2609b35786c186c89396f830ddb6722c"} Sep 10 15:42:40.346395 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:40.346360 2581 patch_prober.go:28] interesting pod/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:42:40.346573 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:40.346419 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:42:40.493088 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:40.493057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" event={"ID":"b953aef1-1fd7-4744-bca2-0bd1b234d35a","Type":"ContainerStarted","Data":"0f5f9f778fae23f247262ad557d8f9a47badff10ce8578b052372e3ed5b74f82"} Sep 10 15:42:40.513813 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:40.513759 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" podStartSLOduration=2.513742817 podStartE2EDuration="2.513742817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:40.510543928 +0000 UTC m=+930.293756992" watchObservedRunningTime="2026-09-10 15:42:40.513742817 +0000 UTC m=+930.296955880" Sep 10 15:42:41.346700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:41.346663 2581 patch_prober.go:28] interesting pod/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:42:41.346873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:41.346721 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:42:41.501227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:41.501184 2581 generic.go:356] "Generic (PLEG): container finished" podID="b953aef1-1fd7-4744-bca2-0bd1b234d35a" containerID="0f5f9f778fae23f247262ad557d8f9a47badff10ce8578b052372e3ed5b74f82" exitCode=0 Sep 10 15:42:41.501623 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:41.501267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" event={"ID":"b953aef1-1fd7-4744-bca2-0bd1b234d35a","Type":"ContainerDied","Data":"0f5f9f778fae23f247262ad557d8f9a47badff10ce8578b052372e3ed5b74f82"} Sep 10 15:42:42.350277 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.350245 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:42.350627 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.350607 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:42.351337 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.351321 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:42.669503 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.669115 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:42.830507 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.830472 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b953aef1-1fd7-4744-bca2-0bd1b234d35a-maas-api-service-ca\") pod \"b953aef1-1fd7-4744-bca2-0bd1b234d35a\" (UID: \"b953aef1-1fd7-4744-bca2-0bd1b234d35a\") " Sep 10 15:42:42.830805 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.830784 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b953aef1-1fd7-4744-bca2-0bd1b234d35a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b953aef1-1fd7-4744-bca2-0bd1b234d35a" (UID: "b953aef1-1fd7-4744-bca2-0bd1b234d35a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:42:42.830890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:42.830877 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b953aef1-1fd7-4744-bca2-0bd1b234d35a-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:43.517898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:43.517871 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" Sep 10 15:42:43.518104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:43.517915 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7" event={"ID":"b953aef1-1fd7-4744-bca2-0bd1b234d35a","Type":"ContainerDied","Data":"945d48ed071001596a5403ac060c8b9c2609b35786c186c89396f830ddb6722c"} Sep 10 15:42:43.518104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:43.517960 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945d48ed071001596a5403ac060c8b9c2609b35786c186c89396f830ddb6722c" Sep 10 15:42:44.678740 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:44.678698 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7"] Sep 10 15:42:44.685082 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:44.685046 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f89c4f3a-xvcn7"] Sep 10 15:42:44.792401 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:44.792360 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b953aef1-1fd7-4744-bca2-0bd1b234d35a" path="/var/lib/kubelet/pods/b953aef1-1fd7-4744-bca2-0bd1b234d35a/volumes" Sep 10 15:42:44.809696 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:44.809655 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4"] Sep 10 15:42:44.809949 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:44.809922 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" containerID="cri-o://c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac" gracePeriod=30 Sep 10 15:42:50.078386 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.078323 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:50.206761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.206727 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-data\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.206946 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.206813 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-envoy\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.206946 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.206859 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75607121-461e-460f-a059-4c10302a323c-istiod-ca-cert\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.206946 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.206899 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-istio-token\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.207177 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.206959 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-certs\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.207177 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207023 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl2hs\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-kube-api-access-nl2hs\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.207177 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207082 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75607121-461e-460f-a059-4c10302a323c-istio-podinfo\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.207177 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207118 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-credential-socket\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.207177 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207150 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-socket\") pod \"75607121-461e-460f-a059-4c10302a323c\" (UID: \"75607121-461e-460f-a059-4c10302a323c\") " Sep 10 15:42:50.207430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207081 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-data" (OuterVolumeSpecName: "istio-data") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:50.207430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207282 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:50.207430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207375 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75607121-461e-460f-a059-4c10302a323c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:42:50.207590 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207479 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:50.207590 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207487 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:50.207590 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207578 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.207743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207598 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75607121-461e-460f-a059-4c10302a323c-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.207743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207613 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.207743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207625 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.207743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.207640 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.209216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.209186 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-kube-api-access-nl2hs" (OuterVolumeSpecName: "kube-api-access-nl2hs") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "kube-api-access-nl2hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:42:50.209327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.209218 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:42:50.209327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.209285 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75607121-461e-460f-a059-4c10302a323c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:42:50.209327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.209289 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-istio-token" (OuterVolumeSpecName: "istio-token") pod "75607121-461e-460f-a059-4c10302a323c" (UID: "75607121-461e-460f-a059-4c10302a323c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:42:50.308970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.308917 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.308970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.308964 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2hs\" (UniqueName: \"kubernetes.io/projected/75607121-461e-460f-a059-4c10302a323c-kube-api-access-nl2hs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.308970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.308977 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75607121-461e-460f-a059-4c10302a323c-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.308970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.308986 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75607121-461e-460f-a059-4c10302a323c-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:42:50.564011 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.563960 2581 generic.go:356] "Generic (PLEG): container finished" podID="75607121-461e-460f-a059-4c10302a323c" containerID="c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac" exitCode=0 Sep 10 15:42:50.564182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.564030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" event={"ID":"75607121-461e-460f-a059-4c10302a323c","Type":"ContainerDied","Data":"c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac"} Sep 10 15:42:50.564182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.564084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" event={"ID":"75607121-461e-460f-a059-4c10302a323c","Type":"ContainerDied","Data":"988be577fb850b203f8d4d9bd5018bc1109acdf5e231758aef7a26066e1aa227"} Sep 10 15:42:50.564182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.564108 2581 scope.go:117] "RemoveContainer" containerID="c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac" Sep 10 15:42:50.564182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.564040 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4" Sep 10 15:42:50.573889 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.573871 2581 scope.go:117] "RemoveContainer" containerID="ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718" Sep 10 15:42:50.581974 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.581953 2581 scope.go:117] "RemoveContainer" containerID="c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac" Sep 10 15:42:50.582261 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:42:50.582240 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac\": container with ID starting with c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac not found: ID does not exist" containerID="c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac" Sep 10 15:42:50.582309 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.582271 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac"} err="failed to get container status \"c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac\": rpc error: code = NotFound desc = could not find container \"c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac\": container with ID starting with c4fa6d21a824af1a776c0a92b4d5231111cc42d3a7bace4b905d84a6c1645cac not found: ID does not exist" Sep 10 15:42:50.582309 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.582288 2581 scope.go:117] "RemoveContainer" containerID="ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718" Sep 10 15:42:50.582504 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:42:50.582488 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718\": container with ID starting with ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718 not found: ID does not exist" containerID="ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718" Sep 10 15:42:50.582543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.582509 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718"} err="failed to get container status \"ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718\": rpc error: code = NotFound desc = could not find container \"ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718\": container with ID starting with ebe8e3b1c5600b0bd4df6a0d401a810511ef0e3d554282eb29ffc5397a833718 not found: ID does not exist" Sep 10 15:42:50.587825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.587804 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4"] Sep 10 15:42:50.602659 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.602632 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f89c4f3a-gw-openshift-default-67f8ff4d8d-nklv4"] Sep 10 15:42:50.790112 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:50.790075 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75607121-461e-460f-a059-4c10302a323c" path="/var/lib/kubelet/pods/75607121-461e-460f-a059-4c10302a323c/volumes" Sep 10 15:42:51.943865 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.943833 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75"] Sep 10 15:42:51.944324 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944301 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b953aef1-1fd7-4744-bca2-0bd1b234d35a" containerName="revoke-keys" Sep 10 15:42:51.944389 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944325 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b953aef1-1fd7-4744-bca2-0bd1b234d35a" containerName="revoke-keys" Sep 10 15:42:51.944389 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944343 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" Sep 10 15:42:51.944389 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944351 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" Sep 10 15:42:51.944389 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944372 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" Sep 10 15:42:51.944389 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944378 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" Sep 10 15:42:51.944543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944445 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" Sep 10 15:42:51.944543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944455 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75607121-461e-460f-a059-4c10302a323c" containerName="istio-proxy" Sep 10 15:42:51.944543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.944461 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b953aef1-1fd7-4744-bca2-0bd1b234d35a" containerName="revoke-keys" Sep 10 15:42:51.947742 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.947726 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:51.950413 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.950388 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bd8496a8-dockercfg-d7fmq\"" Sep 10 15:42:51.950616 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.950441 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bd8496a8\"" Sep 10 15:42:51.961964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:51.961935 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75"] Sep 10 15:42:52.127296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.127259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec0562-96c5-4693-8c32-d4568b6466d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.127475 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.127338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlc4\" (UniqueName: \"kubernetes.io/projected/cbec0562-96c5-4693-8c32-d4568b6466d2-kube-api-access-qvlc4\") pod \"payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.228446 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.228343 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec0562-96c5-4693-8c32-d4568b6466d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.228446 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.228407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlc4\" (UniqueName: \"kubernetes.io/projected/cbec0562-96c5-4693-8c32-d4568b6466d2-kube-api-access-qvlc4\") pod \"payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.229070 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.229048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec0562-96c5-4693-8c32-d4568b6466d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.237602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.237582 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlc4\" (UniqueName: \"kubernetes.io/projected/cbec0562-96c5-4693-8c32-d4568b6466d2-kube-api-access-qvlc4\") pod \"payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.262544 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.262509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:52.537714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.537688 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75"] Sep 10 15:42:52.540240 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:42:52.540207 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbec0562_96c5_4693_8c32_d4568b6466d2.slice/crio-fb332ea7af96808c76bf36d9f4adda0c1558b2173f582ce43a96156a89b340ff WatchSource:0}: Error finding container fb332ea7af96808c76bf36d9f4adda0c1558b2173f582ce43a96156a89b340ff: Status 404 returned error can't find the container with id fb332ea7af96808c76bf36d9f4adda0c1558b2173f582ce43a96156a89b340ff Sep 10 15:42:52.578019 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:52.577969 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" event={"ID":"cbec0562-96c5-4693-8c32-d4568b6466d2","Type":"ContainerStarted","Data":"fb332ea7af96808c76bf36d9f4adda0c1558b2173f582ce43a96156a89b340ff"} Sep 10 15:42:53.585304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:53.585261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" event={"ID":"cbec0562-96c5-4693-8c32-d4568b6466d2","Type":"ContainerStarted","Data":"8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5"} Sep 10 15:42:53.585737 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:53.585411 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:42:53.605174 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:42:53.605120 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" podStartSLOduration=2.60510342 podStartE2EDuration="2.60510342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:53.604883726 +0000 UTC m=+943.388096792" watchObservedRunningTime="2026-09-10 15:42:53.60510342 +0000 UTC m=+943.388316489" Sep 10 15:43:04.593664 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:04.593632 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:43:18.927868 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:18.927828 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc"] Sep 10 15:43:18.933149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:18.933131 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:18.935941 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:18.935917 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:43:18.936088 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:18.935918 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:43:18.952907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:18.952882 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc"] Sep 10 15:43:19.098420 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:19.098378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d2d72f-7226-4ac7-9866-b69567933e02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc\" (UID: \"d3d2d72f-7226-4ac7-9866-b69567933e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:19.200052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:19.199941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d2d72f-7226-4ac7-9866-b69567933e02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc\" (UID: \"d3d2d72f-7226-4ac7-9866-b69567933e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:19.200610 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:19.200587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d2d72f-7226-4ac7-9866-b69567933e02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc\" (UID: \"d3d2d72f-7226-4ac7-9866-b69567933e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:19.243463 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:19.243429 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:19.531465 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:19.531441 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc"] Sep 10 15:43:19.533429 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:43:19.533400 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d2d72f_7226_4ac7_9866_b69567933e02.slice/crio-279f1be780c5d36a8a16f46947b79104a9f10755a505c3eb7827745d3c802cfe WatchSource:0}: Error finding container 279f1be780c5d36a8a16f46947b79104a9f10755a505c3eb7827745d3c802cfe: Status 404 returned error can't find the container with id 279f1be780c5d36a8a16f46947b79104a9f10755a505c3eb7827745d3c802cfe Sep 10 15:43:19.737250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:19.737211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" event={"ID":"d3d2d72f-7226-4ac7-9866-b69567933e02","Type":"ContainerStarted","Data":"279f1be780c5d36a8a16f46947b79104a9f10755a505c3eb7827745d3c802cfe"} Sep 10 15:43:20.744149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:20.744055 2581 generic.go:356] "Generic (PLEG): container finished" podID="d3d2d72f-7226-4ac7-9866-b69567933e02" containerID="728206c6c03adc14f4f2da8542ef4315ad54f0184cc713f36f748130148e6b67" exitCode=0 Sep 10 15:43:20.744149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:20.744115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" event={"ID":"d3d2d72f-7226-4ac7-9866-b69567933e02","Type":"ContainerDied","Data":"728206c6c03adc14f4f2da8542ef4315ad54f0184cc713f36f748130148e6b67"} Sep 10 15:43:21.890558 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:21.890532 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:22.029271 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:22.029185 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d2d72f-7226-4ac7-9866-b69567933e02-maas-api-service-ca\") pod \"d3d2d72f-7226-4ac7-9866-b69567933e02\" (UID: \"d3d2d72f-7226-4ac7-9866-b69567933e02\") " Sep 10 15:43:22.029519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:22.029497 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d2d72f-7226-4ac7-9866-b69567933e02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3d2d72f-7226-4ac7-9866-b69567933e02" (UID: "d3d2d72f-7226-4ac7-9866-b69567933e02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:22.130422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:22.130384 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d2d72f-7226-4ac7-9866-b69567933e02-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:43:22.755876 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:22.755846 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" Sep 10 15:43:22.756064 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:22.755890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc" event={"ID":"d3d2d72f-7226-4ac7-9866-b69567933e02","Type":"ContainerDied","Data":"279f1be780c5d36a8a16f46947b79104a9f10755a505c3eb7827745d3c802cfe"} Sep 10 15:43:22.756064 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:22.755936 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="279f1be780c5d36a8a16f46947b79104a9f10755a505c3eb7827745d3c802cfe" Sep 10 15:43:30.359549 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.359511 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75"] Sep 10 15:43:30.359911 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.359760 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" podUID="cbec0562-96c5-4693-8c32-d4568b6466d2" containerName="payload-processing" containerID="cri-o://8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5" gracePeriod=30 Sep 10 15:43:30.622191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.622132 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:43:30.687350 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.687315 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc"] Sep 10 15:43:30.692442 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.692419 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd8496a8-s6ccc"] Sep 10 15:43:30.717490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.717463 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvlc4\" (UniqueName: \"kubernetes.io/projected/cbec0562-96c5-4693-8c32-d4568b6466d2-kube-api-access-qvlc4\") pod \"cbec0562-96c5-4693-8c32-d4568b6466d2\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " Sep 10 15:43:30.717630 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.717559 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec0562-96c5-4693-8c32-d4568b6466d2-plugins-config-volume\") pod \"cbec0562-96c5-4693-8c32-d4568b6466d2\" (UID: \"cbec0562-96c5-4693-8c32-d4568b6466d2\") " Sep 10 15:43:30.717951 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.717919 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbec0562-96c5-4693-8c32-d4568b6466d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbec0562-96c5-4693-8c32-d4568b6466d2" (UID: "cbec0562-96c5-4693-8c32-d4568b6466d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:30.719692 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.719674 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbec0562-96c5-4693-8c32-d4568b6466d2-kube-api-access-qvlc4" (OuterVolumeSpecName: "kube-api-access-qvlc4") pod "cbec0562-96c5-4693-8c32-d4568b6466d2" (UID: "cbec0562-96c5-4693-8c32-d4568b6466d2"). InnerVolumeSpecName "kube-api-access-qvlc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:43:30.790160 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.790126 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d2d72f-7226-4ac7-9866-b69567933e02" path="/var/lib/kubelet/pods/d3d2d72f-7226-4ac7-9866-b69567933e02/volumes" Sep 10 15:43:30.808346 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.808313 2581 generic.go:356] "Generic (PLEG): container finished" podID="cbec0562-96c5-4693-8c32-d4568b6466d2" containerID="8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5" exitCode=0 Sep 10 15:43:30.808474 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.808369 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" Sep 10 15:43:30.808474 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.808405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" event={"ID":"cbec0562-96c5-4693-8c32-d4568b6466d2","Type":"ContainerDied","Data":"8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5"} Sep 10 15:43:30.808558 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.808485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75" event={"ID":"cbec0562-96c5-4693-8c32-d4568b6466d2","Type":"ContainerDied","Data":"fb332ea7af96808c76bf36d9f4adda0c1558b2173f582ce43a96156a89b340ff"} Sep 10 15:43:30.808558 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.808509 2581 scope.go:117] "RemoveContainer" containerID="8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5" Sep 10 15:43:30.817435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.817415 2581 scope.go:117] "RemoveContainer" containerID="8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5" Sep 10 15:43:30.817727 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:43:30.817709 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5\": container with ID starting with 8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5 not found: ID does not exist" containerID="8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5" Sep 10 15:43:30.817811 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.817733 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5"} err="failed to get container status \"8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5\": rpc error: code = NotFound desc = could not find container \"8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5\": container with ID starting with 8af038e9a37c1d940789991b607014de7817652ea395fed361e1e21d7806efc5 not found: ID does not exist" Sep 10 15:43:30.819398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.819373 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlc4\" (UniqueName: \"kubernetes.io/projected/cbec0562-96c5-4693-8c32-d4568b6466d2-kube-api-access-qvlc4\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:43:30.819488 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.819401 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbec0562-96c5-4693-8c32-d4568b6466d2-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:43:30.828187 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.828166 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75"] Sep 10 15:43:30.836289 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:30.836254 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd8496a8-6f65f854c9-9pd75"] Sep 10 15:43:32.764965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.764929 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp"] Sep 10 15:43:32.765373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.765360 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbec0562-96c5-4693-8c32-d4568b6466d2" containerName="payload-processing" Sep 10 15:43:32.765373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.765375 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbec0562-96c5-4693-8c32-d4568b6466d2" containerName="payload-processing" Sep 10 15:43:32.765449 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.765396 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d2d72f-7226-4ac7-9866-b69567933e02" containerName="revoke-keys" Sep 10 15:43:32.765449 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.765402 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d2d72f-7226-4ac7-9866-b69567933e02" containerName="revoke-keys" Sep 10 15:43:32.765509 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.765466 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d2d72f-7226-4ac7-9866-b69567933e02" containerName="revoke-keys" Sep 10 15:43:32.765509 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.765476 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbec0562-96c5-4693-8c32-d4568b6466d2" containerName="payload-processing" Sep 10 15:43:32.769950 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.769932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.774261 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.774235 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-680721f9-openshift-default-dockercfg-hlz9h\"" Sep 10 15:43:32.788735 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.788705 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbec0562-96c5-4693-8c32-d4568b6466d2" path="/var/lib/kubelet/pods/cbec0562-96c5-4693-8c32-d4568b6466d2/volumes" Sep 10 15:43:32.804127 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.804096 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp"] Sep 10 15:43:32.839505 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839481 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e892e7-70b5-452a-b692-9c69da28944d-istio-podinfo\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-socket\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-credential-socket\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-envoy\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2phg\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-kube-api-access-l2phg\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e892e7-70b5-452a-b692-9c69da28944d-istiod-ca-cert\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839755 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-data\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-istio-token\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.839930 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.839807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-certs\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.940883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.940796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e892e7-70b5-452a-b692-9c69da28944d-istio-podinfo\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.940883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.940852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-socket\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.940896 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-credential-socket\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.940922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-envoy\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2phg\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-kube-api-access-l2phg\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e892e7-70b5-452a-b692-9c69da28944d-istiod-ca-cert\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-data\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941155 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-istio-token\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-certs\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941359 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-credential-socket\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-socket\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941465 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-data\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-certs\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.941760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.941713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e892e7-70b5-452a-b692-9c69da28944d-istiod-ca-cert\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.943243 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.943222 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-envoy\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.943574 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.943560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e892e7-70b5-452a-b692-9c69da28944d-istio-podinfo\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.953045 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.952982 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-istio-token\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:32.953146 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:32.953108 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2phg\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-kube-api-access-l2phg\") pod \"e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:33.080092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.080060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:33.362305 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.362277 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp"] Sep 10 15:43:33.363938 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:43:33.363912 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e892e7_70b5_452a_b692_9c69da28944d.slice/crio-9b190cd0ae597268bcbd9a80e04d085cebe36aefb6053abf80bc93f721e06660 WatchSource:0}: Error finding container 9b190cd0ae597268bcbd9a80e04d085cebe36aefb6053abf80bc93f721e06660: Status 404 returned error can't find the container with id 9b190cd0ae597268bcbd9a80e04d085cebe36aefb6053abf80bc93f721e06660 Sep 10 15:43:33.366079 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.366040 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:43:33.366148 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.366119 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:43:33.366188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.366146 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:43:33.829852 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.829813 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" event={"ID":"e7e892e7-70b5-452a-b692-9c69da28944d","Type":"ContainerStarted","Data":"3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4"} Sep 10 15:43:33.830248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.829881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" event={"ID":"e7e892e7-70b5-452a-b692-9c69da28944d","Type":"ContainerStarted","Data":"9b190cd0ae597268bcbd9a80e04d085cebe36aefb6053abf80bc93f721e06660"} Sep 10 15:43:33.869172 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:33.869118 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podStartSLOduration=1.86910279 podStartE2EDuration="1.86910279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:33.868524204 +0000 UTC m=+983.651737269" watchObservedRunningTime="2026-09-10 15:43:33.86910279 +0000 UTC m=+983.652315854" Sep 10 15:43:34.080322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:34.080218 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:34.085133 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:34.085110 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:34.836077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:34.836048 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:34.837042 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:34.837019 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:43:40.817220 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.817186 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2"] Sep 10 15:43:40.821006 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.820976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:40.830395 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.830371 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-680721f9-dockercfg-4sfqk\"" Sep 10 15:43:40.830882 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.830862 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-680721f9\"" Sep 10 15:43:40.836027 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.835979 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2"] Sep 10 15:43:40.887793 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.887759 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5"] Sep 10 15:43:40.891528 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.891507 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:40.915135 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.915108 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5"] Sep 10 15:43:40.919535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.919506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:40.919719 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:40.919697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c6d\" (UniqueName: \"kubernetes.io/projected/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-kube-api-access-24c6d\") pod \"payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.020735 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.020696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2612b0e-3a79-406a-b549-e3c89fbb5943-plugins-config-volume\") pod \"payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.020931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.020756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c6d\" (UniqueName: \"kubernetes.io/projected/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-kube-api-access-24c6d\") pod \"payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.020931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.020837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdcz\" (UniqueName: \"kubernetes.io/projected/a2612b0e-3a79-406a-b549-e3c89fbb5943-kube-api-access-6fdcz\") pod \"payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.020931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.020902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.021588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.021569 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.031391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.031364 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c6d\" (UniqueName: \"kubernetes.io/projected/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-kube-api-access-24c6d\") pod \"payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.122288 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.122194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2612b0e-3a79-406a-b549-e3c89fbb5943-plugins-config-volume\") pod \"payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.122288 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.122262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdcz\" (UniqueName: \"kubernetes.io/projected/a2612b0e-3a79-406a-b549-e3c89fbb5943-kube-api-access-6fdcz\") pod \"payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.122937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.122907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2612b0e-3a79-406a-b549-e3c89fbb5943-plugins-config-volume\") pod \"payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.132100 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.132079 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdcz\" (UniqueName: \"kubernetes.io/projected/a2612b0e-3a79-406a-b549-e3c89fbb5943-kube-api-access-6fdcz\") pod \"payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.134915 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.134895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.201154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.201125 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.412832 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.412803 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2"] Sep 10 15:43:41.413809 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:43:41.413784 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ec2e50_63fb_4473_a9c2_e3eb1ae575be.slice/crio-fa2daf9032c371e07c5fe9ce440755ceb2118635148efe71eb9579f5bf008d20 WatchSource:0}: Error finding container fa2daf9032c371e07c5fe9ce440755ceb2118635148efe71eb9579f5bf008d20: Status 404 returned error can't find the container with id fa2daf9032c371e07c5fe9ce440755ceb2118635148efe71eb9579f5bf008d20 Sep 10 15:43:41.480640 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.480609 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5"] Sep 10 15:43:41.481256 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:43:41.481231 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2612b0e_3a79_406a_b549_e3c89fbb5943.slice/crio-41d08f5093572b77ed28bc920a393a276c8b9eee94c916eb347e1c9fb2fb4892 WatchSource:0}: Error finding container 41d08f5093572b77ed28bc920a393a276c8b9eee94c916eb347e1c9fb2fb4892: Status 404 returned error can't find the container with id 41d08f5093572b77ed28bc920a393a276c8b9eee94c916eb347e1c9fb2fb4892 Sep 10 15:43:41.880299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.880256 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" event={"ID":"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be","Type":"ContainerStarted","Data":"df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a"} Sep 10 15:43:41.880777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.880328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" event={"ID":"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be","Type":"ContainerStarted","Data":"fa2daf9032c371e07c5fe9ce440755ceb2118635148efe71eb9579f5bf008d20"} Sep 10 15:43:41.880777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.880374 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:41.882006 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.881975 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" event={"ID":"a2612b0e-3a79-406a-b549-e3c89fbb5943","Type":"ContainerStarted","Data":"1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963"} Sep 10 15:43:41.882110 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.882036 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:41.882110 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.882053 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" event={"ID":"a2612b0e-3a79-406a-b549-e3c89fbb5943","Type":"ContainerStarted","Data":"41d08f5093572b77ed28bc920a393a276c8b9eee94c916eb347e1c9fb2fb4892"} Sep 10 15:43:41.902148 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.902101 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" podStartSLOduration=1.90208749 podStartE2EDuration="1.90208749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:41.899402801 +0000 UTC m=+991.682615865" watchObservedRunningTime="2026-09-10 15:43:41.90208749 +0000 UTC m=+991.685300617" Sep 10 15:43:41.918949 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:41.918906 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" podStartSLOduration=1.91889223 podStartE2EDuration="1.91889223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:41.917370156 +0000 UTC m=+991.700583220" watchObservedRunningTime="2026-09-10 15:43:41.91889223 +0000 UTC m=+991.702105293" Sep 10 15:43:43.418645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.418613 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm"] Sep 10 15:43:43.422363 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.422347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:43.424910 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.424884 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:43:43.425057 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.424923 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:43:43.431022 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.430984 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm"] Sep 10 15:43:43.551810 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.551774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bd5806-r2mpm\" (UID: \"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:43.653011 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.652958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bd5806-r2mpm\" (UID: \"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:43.653579 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.653558 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bd5806-r2mpm\" (UID: \"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:43.734480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:43.734393 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:44.014907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:44.014818 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm"] Sep 10 15:43:44.022145 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:43:44.022113 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf4d1c5_fc5a_4594_9bd4_30ddde5698a3.slice/crio-bd28669376181686d35dd86b70599dbffc19abc7dbca68655b4cdaf1216c4ab8 WatchSource:0}: Error finding container bd28669376181686d35dd86b70599dbffc19abc7dbca68655b4cdaf1216c4ab8: Status 404 returned error can't find the container with id bd28669376181686d35dd86b70599dbffc19abc7dbca68655b4cdaf1216c4ab8 Sep 10 15:43:44.900622 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:44.900584 2581 generic.go:356] "Generic (PLEG): container finished" podID="faf4d1c5-fc5a-4594-9bd4-30ddde5698a3" containerID="c54c559f1f324c50e4827cf5d5b99f53f12ed748ad4b9978d16fa96d881074d6" exitCode=0 Sep 10 15:43:44.900964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:44.900670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" event={"ID":"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3","Type":"ContainerDied","Data":"c54c559f1f324c50e4827cf5d5b99f53f12ed748ad4b9978d16fa96d881074d6"} Sep 10 15:43:44.900964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:44.900726 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" event={"ID":"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3","Type":"ContainerStarted","Data":"bd28669376181686d35dd86b70599dbffc19abc7dbca68655b4cdaf1216c4ab8"} Sep 10 15:43:46.043121 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.043100 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:46.178905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.178820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3-maas-api-service-ca\") pod \"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3\" (UID: \"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3\") " Sep 10 15:43:46.179247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.179225 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "faf4d1c5-fc5a-4594-9bd4-30ddde5698a3" (UID: "faf4d1c5-fc5a-4594-9bd4-30ddde5698a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:46.280410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.280375 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:43:46.913502 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.913471 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" event={"ID":"faf4d1c5-fc5a-4594-9bd4-30ddde5698a3","Type":"ContainerDied","Data":"bd28669376181686d35dd86b70599dbffc19abc7dbca68655b4cdaf1216c4ab8"} Sep 10 15:43:46.913662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.913524 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd28669376181686d35dd86b70599dbffc19abc7dbca68655b4cdaf1216c4ab8" Sep 10 15:43:46.913662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:46.913481 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm" Sep 10 15:43:52.890753 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:52.890717 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:43:52.891184 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:52.890857 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:43:54.230791 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.230756 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g"] Sep 10 15:43:54.231305 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.230982 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" podUID="7b9276e3-474d-4014-96b6-0d84030543c1" containerName="payload-pre-processing" containerID="cri-o://8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f" gracePeriod=30 Sep 10 15:43:54.432883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.432852 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm"] Sep 10 15:43:54.441282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.441252 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bd5806-r2mpm"] Sep 10 15:43:54.540230 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.540202 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:43:54.687202 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.687171 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2rv4\" (UniqueName: \"kubernetes.io/projected/7b9276e3-474d-4014-96b6-0d84030543c1-kube-api-access-q2rv4\") pod \"7b9276e3-474d-4014-96b6-0d84030543c1\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " Sep 10 15:43:54.687397 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.687267 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9276e3-474d-4014-96b6-0d84030543c1-plugins-config-volume\") pod \"7b9276e3-474d-4014-96b6-0d84030543c1\" (UID: \"7b9276e3-474d-4014-96b6-0d84030543c1\") " Sep 10 15:43:54.687639 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.687614 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9276e3-474d-4014-96b6-0d84030543c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b9276e3-474d-4014-96b6-0d84030543c1" (UID: "7b9276e3-474d-4014-96b6-0d84030543c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:43:54.689398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.689360 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9276e3-474d-4014-96b6-0d84030543c1-kube-api-access-q2rv4" (OuterVolumeSpecName: "kube-api-access-q2rv4") pod "7b9276e3-474d-4014-96b6-0d84030543c1" (UID: "7b9276e3-474d-4014-96b6-0d84030543c1"). InnerVolumeSpecName "kube-api-access-q2rv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:43:54.788318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.788236 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9276e3-474d-4014-96b6-0d84030543c1-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:43:54.788318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.788260 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rv4\" (UniqueName: \"kubernetes.io/projected/7b9276e3-474d-4014-96b6-0d84030543c1-kube-api-access-q2rv4\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:43:54.789160 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.789137 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faf4d1c5-fc5a-4594-9bd4-30ddde5698a3" path="/var/lib/kubelet/pods/faf4d1c5-fc5a-4594-9bd4-30ddde5698a3/volumes" Sep 10 15:43:54.985409 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.985374 2581 generic.go:356] "Generic (PLEG): container finished" podID="7b9276e3-474d-4014-96b6-0d84030543c1" containerID="8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f" exitCode=0 Sep 10 15:43:54.985562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.985436 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" Sep 10 15:43:54.985562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.985447 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" event={"ID":"7b9276e3-474d-4014-96b6-0d84030543c1","Type":"ContainerDied","Data":"8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f"} Sep 10 15:43:54.985562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.985513 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g" event={"ID":"7b9276e3-474d-4014-96b6-0d84030543c1","Type":"ContainerDied","Data":"ab032f87b04d78aaabf6fcedcc96619478fd343fc21465f4e66de46d3f970757"} Sep 10 15:43:54.985562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.985534 2581 scope.go:117] "RemoveContainer" containerID="8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f" Sep 10 15:43:54.994465 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.994447 2581 scope.go:117] "RemoveContainer" containerID="8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f" Sep 10 15:43:54.994722 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:43:54.994706 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f\": container with ID starting with 8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f not found: ID does not exist" containerID="8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f" Sep 10 15:43:54.994759 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:54.994729 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f"} err="failed to get container status \"8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f\": rpc error: code = NotFound desc = could not find container \"8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f\": container with ID starting with 8e13996c8d98d9e10256df22f7dd4d3faa353be4fb6a0f4da3fde3c2c1303a1f not found: ID does not exist" Sep 10 15:43:55.004220 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:55.004192 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g"] Sep 10 15:43:55.008666 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:55.008641 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bd5806-5666dc795d-nxc4g"] Sep 10 15:43:56.789782 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:43:56.789691 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b9276e3-474d-4014-96b6-0d84030543c1" path="/var/lib/kubelet/pods/7b9276e3-474d-4014-96b6-0d84030543c1/volumes" Sep 10 15:44:00.016610 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.016575 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7e892e7-70b5-452a-b692-9c69da28944d" containerID="3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4" exitCode=0 Sep 10 15:44:00.017039 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.016647 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" event={"ID":"e7e892e7-70b5-452a-b692-9c69da28944d","Type":"ContainerDied","Data":"3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4"} Sep 10 15:44:00.017165 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.017148 2581 scope.go:117] "RemoveContainer" containerID="3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4" Sep 10 15:44:00.018262 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.018234 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:44:00.018330 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.018307 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:44:00.018367 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.018341 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:44:00.211297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.211261 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4"] Sep 10 15:44:00.211918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.211903 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf4d1c5-fc5a-4594-9bd4-30ddde5698a3" containerName="revoke-keys" Sep 10 15:44:00.211970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.211921 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf4d1c5-fc5a-4594-9bd4-30ddde5698a3" containerName="revoke-keys" Sep 10 15:44:00.211970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.211954 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9276e3-474d-4014-96b6-0d84030543c1" containerName="payload-pre-processing" Sep 10 15:44:00.211970 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.211962 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9276e3-474d-4014-96b6-0d84030543c1" containerName="payload-pre-processing" Sep 10 15:44:00.212111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.212089 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf4d1c5-fc5a-4594-9bd4-30ddde5698a3" containerName="revoke-keys" Sep 10 15:44:00.212111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.212103 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9276e3-474d-4014-96b6-0d84030543c1" containerName="payload-pre-processing" Sep 10 15:44:00.214729 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.214709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:00.220244 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.220207 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:44:00.220400 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.220265 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:44:00.225663 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.225637 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4"] Sep 10 15:44:00.244713 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.244440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-680721f9-gqmk4\" (UID: \"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:00.346297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.346266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-680721f9-gqmk4\" (UID: \"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:00.346982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.346955 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-680721f9-gqmk4\" (UID: \"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:00.528251 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.528213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:00.800320 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:00.800297 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4"] Sep 10 15:44:00.802500 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:44:00.802472 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322ba4c4_a944_4de6_ab0f_d7a5078ec7b2.slice/crio-9924d0a38c83551949695ab669a27f3dc66789141b830d2606c8a8c4c689eea2 WatchSource:0}: Error finding container 9924d0a38c83551949695ab669a27f3dc66789141b830d2606c8a8c4c689eea2: Status 404 returned error can't find the container with id 9924d0a38c83551949695ab669a27f3dc66789141b830d2606c8a8c4c689eea2 Sep 10 15:44:01.027380 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:01.027344 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" event={"ID":"e7e892e7-70b5-452a-b692-9c69da28944d","Type":"ContainerStarted","Data":"863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9"} Sep 10 15:44:01.028654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:01.028628 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" event={"ID":"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2","Type":"ContainerStarted","Data":"9924d0a38c83551949695ab669a27f3dc66789141b830d2606c8a8c4c689eea2"} Sep 10 15:44:01.080463 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:01.080437 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:44:01.081611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:01.081589 2581 patch_prober.go:28] interesting pod/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:44:01.081676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:01.081632 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 10 15:44:02.036048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:02.035943 2581 generic.go:356] "Generic (PLEG): container finished" podID="322ba4c4-a944-4de6-ab0f-d7a5078ec7b2" containerID="40b055869c3d4e5e8c5ef8a50453a4913dbc05382294619a7760a140cc14e82e" exitCode=0 Sep 10 15:44:02.036048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:02.036016 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" event={"ID":"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2","Type":"ContainerDied","Data":"40b055869c3d4e5e8c5ef8a50453a4913dbc05382294619a7760a140cc14e82e"} Sep 10 15:44:02.080729 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:02.080689 2581 patch_prober.go:28] interesting pod/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:44:02.080904 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:02.080749 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 10 15:44:03.080827 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:03.080786 2581 patch_prober.go:28] interesting pod/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:44:03.081295 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:03.080843 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 10 15:44:03.240727 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:03.240700 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:03.280951 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:03.280594 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2-maas-api-service-ca\") pod \"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2\" (UID: \"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2\") " Sep 10 15:44:03.280951 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:03.280899 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "322ba4c4-a944-4de6-ab0f-d7a5078ec7b2" (UID: "322ba4c4-a944-4de6-ab0f-d7a5078ec7b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:03.281228 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:03.281073 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:04.048427 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:04.048396 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" Sep 10 15:44:04.048599 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:04.048401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4" event={"ID":"322ba4c4-a944-4de6-ab0f-d7a5078ec7b2","Type":"ContainerDied","Data":"9924d0a38c83551949695ab669a27f3dc66789141b830d2606c8a8c4c689eea2"} Sep 10 15:44:04.048599 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:04.048517 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9924d0a38c83551949695ab669a27f3dc66789141b830d2606c8a8c4c689eea2" Sep 10 15:44:04.081417 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:04.081379 2581 patch_prober.go:28] interesting pod/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:44:04.081785 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:04.081444 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 10 15:44:04.836256 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:04.836222 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:44:05.080554 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:05.080521 2581 patch_prober.go:28] interesting pod/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:44:05.080752 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:05.080574 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 10 15:44:06.081369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:06.081332 2581 patch_prober.go:28] interesting pod/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:44:06.081760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:06.081387 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 10 15:44:07.084444 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.084415 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:44:07.085426 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.085405 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:44:07.575338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.575307 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5"] Sep 10 15:44:07.575575 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.575546 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" podUID="a2612b0e-3a79-406a-b549-e3c89fbb5943" containerName="payload-processing" containerID="cri-o://1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963" gracePeriod=30 Sep 10 15:44:07.599283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.599252 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2"] Sep 10 15:44:07.599475 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.599438 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" podUID="b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" containerName="payload-pre-processing" containerID="cri-o://df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a" gracePeriod=30 Sep 10 15:44:07.615876 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.615842 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls"] Sep 10 15:44:07.616151 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.616124 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" containerID="cri-o://76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967" gracePeriod=30 Sep 10 15:44:07.833710 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.833630 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4"] Sep 10 15:44:07.838381 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.838299 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-680721f9-gqmk4"] Sep 10 15:44:07.886971 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.886946 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:44:07.890112 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.890092 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:44:07.928033 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.927959 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24c6d\" (UniqueName: \"kubernetes.io/projected/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-kube-api-access-24c6d\") pod \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " Sep 10 15:44:07.928239 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.928130 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2612b0e-3a79-406a-b549-e3c89fbb5943-plugins-config-volume\") pod \"a2612b0e-3a79-406a-b549-e3c89fbb5943\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " Sep 10 15:44:07.928239 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.928206 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fdcz\" (UniqueName: \"kubernetes.io/projected/a2612b0e-3a79-406a-b549-e3c89fbb5943-kube-api-access-6fdcz\") pod \"a2612b0e-3a79-406a-b549-e3c89fbb5943\" (UID: \"a2612b0e-3a79-406a-b549-e3c89fbb5943\") " Sep 10 15:44:07.928365 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.928253 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-plugins-config-volume\") pod \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\" (UID: \"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be\") " Sep 10 15:44:07.929135 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.929103 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" (UID: "b3ec2e50-63fb-4473-a9c2-e3eb1ae575be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:07.929901 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.929864 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2612b0e-3a79-406a-b549-e3c89fbb5943-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2612b0e-3a79-406a-b549-e3c89fbb5943" (UID: "a2612b0e-3a79-406a-b549-e3c89fbb5943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:07.933446 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.932379 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2612b0e-3a79-406a-b549-e3c89fbb5943-kube-api-access-6fdcz" (OuterVolumeSpecName: "kube-api-access-6fdcz") pod "a2612b0e-3a79-406a-b549-e3c89fbb5943" (UID: "a2612b0e-3a79-406a-b549-e3c89fbb5943"). InnerVolumeSpecName "kube-api-access-6fdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:07.937638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.937610 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-kube-api-access-24c6d" (OuterVolumeSpecName: "kube-api-access-24c6d") pod "b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" (UID: "b3ec2e50-63fb-4473-a9c2-e3eb1ae575be"). InnerVolumeSpecName "kube-api-access-24c6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:07.988883 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:07.988847 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp"] Sep 10 15:44:08.029734 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.029695 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2612b0e-3a79-406a-b549-e3c89fbb5943-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:08.029734 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.029728 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdcz\" (UniqueName: \"kubernetes.io/projected/a2612b0e-3a79-406a-b549-e3c89fbb5943-kube-api-access-6fdcz\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:08.029734 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.029738 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:08.029967 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.029746 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24c6d\" (UniqueName: \"kubernetes.io/projected/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be-kube-api-access-24c6d\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:08.081369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.080166 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" containerID="df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a" exitCode=0 Sep 10 15:44:08.081369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.080225 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" Sep 10 15:44:08.081369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.080330 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" event={"ID":"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be","Type":"ContainerDied","Data":"df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a"} Sep 10 15:44:08.081369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.080389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2" event={"ID":"b3ec2e50-63fb-4473-a9c2-e3eb1ae575be","Type":"ContainerDied","Data":"fa2daf9032c371e07c5fe9ce440755ceb2118635148efe71eb9579f5bf008d20"} Sep 10 15:44:08.081369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.080431 2581 scope.go:117] "RemoveContainer" containerID="df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a" Sep 10 15:44:08.089044 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.087264 2581 generic.go:356] "Generic (PLEG): container finished" podID="a2612b0e-3a79-406a-b549-e3c89fbb5943" containerID="1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963" exitCode=0 Sep 10 15:44:08.089044 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.087814 2581 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" secret="" err="secret \"e2e-derive-680721f9-openshift-default-dockercfg-hlz9h\" not found" Sep 10 15:44:08.089044 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.088354 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" Sep 10 15:44:08.089613 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.089269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" event={"ID":"a2612b0e-3a79-406a-b549-e3c89fbb5943","Type":"ContainerDied","Data":"1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963"} Sep 10 15:44:08.089613 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.089341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5" event={"ID":"a2612b0e-3a79-406a-b549-e3c89fbb5943","Type":"ContainerDied","Data":"41d08f5093572b77ed28bc920a393a276c8b9eee94c916eb347e1c9fb2fb4892"} Sep 10 15:44:08.103797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.100787 2581 scope.go:117] "RemoveContainer" containerID="df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a" Sep 10 15:44:08.103797 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:44:08.101261 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a\": container with ID starting with df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a not found: ID does not exist" containerID="df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a" Sep 10 15:44:08.103797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.101297 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a"} err="failed to get container status \"df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a\": rpc error: code = NotFound desc = could not find container \"df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a\": container with ID starting with df06fcb6e1be5e3176f7bde9777f73515cdd8b3d4fd48b8df8e01e1d75b8e34a not found: ID does not exist" Sep 10 15:44:08.103797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.101322 2581 scope.go:117] "RemoveContainer" containerID="1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963" Sep 10 15:44:08.118301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.116567 2581 scope.go:117] "RemoveContainer" containerID="1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963" Sep 10 15:44:08.118301 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:44:08.117054 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963\": container with ID starting with 1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963 not found: ID does not exist" containerID="1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963" Sep 10 15:44:08.118301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.117099 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963"} err="failed to get container status \"1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963\": rpc error: code = NotFound desc = could not find container \"1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963\": container with ID starting with 1a4aadd680bf5e351f1fc931fe5e4998a6f25b850d2a82e866522f531b083963 not found: ID does not exist" Sep 10 15:44:08.118301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.118180 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2"] Sep 10 15:44:08.120798 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.120759 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-680721f9-5d946c6954-k6bx2"] Sep 10 15:44:08.136761 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.136737 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5"] Sep 10 15:44:08.139744 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.139723 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-680721f9-686c6bdd7c-n9nh5"] Sep 10 15:44:08.790333 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.790297 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322ba4c4-a944-4de6-ab0f-d7a5078ec7b2" path="/var/lib/kubelet/pods/322ba4c4-a944-4de6-ab0f-d7a5078ec7b2/volumes" Sep 10 15:44:08.790855 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.790830 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2612b0e-3a79-406a-b549-e3c89fbb5943" path="/var/lib/kubelet/pods/a2612b0e-3a79-406a-b549-e3c89fbb5943/volumes" Sep 10 15:44:08.791384 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:08.791365 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" path="/var/lib/kubelet/pods/b3ec2e50-63fb-4473-a9c2-e3eb1ae575be/volumes" Sep 10 15:44:09.095690 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:09.095633 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" containerID="cri-o://863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9" gracePeriod=30 Sep 10 15:44:11.604546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:11.604505 2581 patch_prober.go:28] interesting pod/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 15:44:11.604923 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:11.604579 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:44:12.873884 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.873859 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:44:12.979772 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.979680 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-token\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.979772 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.979736 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istiod-ca-cert\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.979887 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-envoy\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.979923 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-certs\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.979954 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j9hj\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-kube-api-access-4j9hj\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980028 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980019 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-podinfo\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980076 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-credential-socket\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980127 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-data\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980162 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-socket\") pod \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\" (UID: \"a96f20ea-2bbd-4fd6-b09b-7c167e990314\") " Sep 10 15:44:12.980249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980170 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:12.980249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980198 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:12.980562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980428 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:12.980562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980448 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-data" (OuterVolumeSpecName: "istio-data") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:12.980662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980601 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:12.980662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980619 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:12.980662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980617 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:12.980662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980633 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:12.980662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.980645 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:12.982107 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.982083 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:12.982454 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.982430 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:44:12.982575 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.982558 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-token" (OuterVolumeSpecName: "istio-token") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:12.982806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:12.982789 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-kube-api-access-4j9hj" (OuterVolumeSpecName: "kube-api-access-4j9hj") pod "a96f20ea-2bbd-4fd6-b09b-7c167e990314" (UID: "a96f20ea-2bbd-4fd6-b09b-7c167e990314"). InnerVolumeSpecName "kube-api-access-4j9hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:13.082077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.082035 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:13.082077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.082071 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:13.082077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.082081 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:13.082077 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.082090 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9hj\" (UniqueName: \"kubernetes.io/projected/a96f20ea-2bbd-4fd6-b09b-7c167e990314-kube-api-access-4j9hj\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:13.082349 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.082100 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96f20ea-2bbd-4fd6-b09b-7c167e990314-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:13.119360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.119327 2581 generic.go:356] "Generic (PLEG): container finished" podID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerID="76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967" exitCode=0 Sep 10 15:44:13.119523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.119377 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" Sep 10 15:44:13.119523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.119420 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" event={"ID":"a96f20ea-2bbd-4fd6-b09b-7c167e990314","Type":"ContainerDied","Data":"76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967"} Sep 10 15:44:13.119523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.119474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls" event={"ID":"a96f20ea-2bbd-4fd6-b09b-7c167e990314","Type":"ContainerDied","Data":"4dcbfb5487394c12f0c5b9272000ed79a77faf331fa1c6317a798299d6be3048"} Sep 10 15:44:13.119523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.119497 2581 scope.go:117] "RemoveContainer" containerID="76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967" Sep 10 15:44:13.129082 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.129064 2581 scope.go:117] "RemoveContainer" containerID="382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1" Sep 10 15:44:13.137698 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.137679 2581 scope.go:117] "RemoveContainer" containerID="76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967" Sep 10 15:44:13.137985 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:44:13.137962 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967\": container with ID starting with 76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967 not found: ID does not exist" containerID="76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967" Sep 10 15:44:13.138068 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.138005 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967"} err="failed to get container status \"76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967\": rpc error: code = NotFound desc = could not find container \"76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967\": container with ID starting with 76a200738bd7079e89187fc5c313c4bb1fb1bcd03e71cb1f28368f28ee142967 not found: ID does not exist" Sep 10 15:44:13.138068 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.138024 2581 scope.go:117] "RemoveContainer" containerID="382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1" Sep 10 15:44:13.138285 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:44:13.138268 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1\": container with ID starting with 382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1 not found: ID does not exist" containerID="382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1" Sep 10 15:44:13.138328 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.138292 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1"} err="failed to get container status \"382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1\": rpc error: code = NotFound desc = could not find container \"382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1\": container with ID starting with 382cb5b1459a5d3cbcf049e56e0c149f78ea06edc6380b17c5586e10395d94f1 not found: ID does not exist" Sep 10 15:44:13.143268 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.143245 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls"] Sep 10 15:44:13.150250 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:13.150229 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-bd5806-openshift-default-54bb46bf6d-n6dls"] Sep 10 15:44:14.360278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.360220 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:44:14.395825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.395798 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-data\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.395825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.395834 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-certs\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396086 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.395863 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-istio-token\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396086 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.395902 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-envoy\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396086 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.395926 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e892e7-70b5-452a-b692-9c69da28944d-istiod-ca-cert\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396086 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.395967 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-socket\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396086 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396015 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-credential-socket\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396086 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396039 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2phg\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-kube-api-access-l2phg\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396383 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396129 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e892e7-70b5-452a-b692-9c69da28944d-istio-podinfo\") pod \"e7e892e7-70b5-452a-b692-9c69da28944d\" (UID: \"e7e892e7-70b5-452a-b692-9c69da28944d\") " Sep 10 15:44:14.396383 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396158 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:14.396535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396382 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e892e7-70b5-452a-b692-9c69da28944d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:14.396535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396388 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:14.396535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396499 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:14.396535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396524 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:14.396739 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396546 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e892e7-70b5-452a-b692-9c69da28944d-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.396739 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396565 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.396739 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.396579 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.398482 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.398453 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-kube-api-access-l2phg" (OuterVolumeSpecName: "kube-api-access-l2phg") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "kube-api-access-l2phg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:14.398787 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.398767 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:14.398856 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.398835 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:14.398905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.398888 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7e892e7-70b5-452a-b692-9c69da28944d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7e892e7-70b5-452a-b692-9c69da28944d" (UID: "e7e892e7-70b5-452a-b692-9c69da28944d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:44:14.497800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.497766 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.497800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.497792 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.497800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.497800 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.497800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.497809 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e892e7-70b5-452a-b692-9c69da28944d-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.498085 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.497817 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2phg\" (UniqueName: \"kubernetes.io/projected/e7e892e7-70b5-452a-b692-9c69da28944d-kube-api-access-l2phg\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.498085 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.497827 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e892e7-70b5-452a-b692-9c69da28944d-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:44:14.788232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.788154 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" path="/var/lib/kubelet/pods/a96f20ea-2bbd-4fd6-b09b-7c167e990314/volumes" Sep 10 15:44:14.898676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.898646 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg"] Sep 10 15:44:14.899102 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899088 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899106 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899118 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" containerName="payload-pre-processing" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899124 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" containerName="payload-pre-processing" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899133 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899139 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899145 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2612b0e-3a79-406a-b549-e3c89fbb5943" containerName="payload-processing" Sep 10 15:44:14.899149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899150 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2612b0e-3a79-406a-b549-e3c89fbb5943" containerName="payload-processing" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899164 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899169 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899192 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322ba4c4-a944-4de6-ab0f-d7a5078ec7b2" containerName="revoke-keys" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899199 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="322ba4c4-a944-4de6-ab0f-d7a5078ec7b2" containerName="revoke-keys" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899268 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2612b0e-3a79-406a-b549-e3c89fbb5943" containerName="payload-processing" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899278 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899283 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899287 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="322ba4c4-a944-4de6-ab0f-d7a5078ec7b2" containerName="revoke-keys" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899294 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ec2e50-63fb-4473-a9c2-e3eb1ae575be" containerName="payload-pre-processing" Sep 10 15:44:14.899362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.899302 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" Sep 10 15:44:14.902531 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.902510 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:14.904912 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.904894 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-fc495e\"" Sep 10 15:44:14.905015 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.904937 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-fc495e-dockercfg-fklvn\"" Sep 10 15:44:14.913450 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.913427 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg"] Sep 10 15:44:14.930853 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.930822 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr"] Sep 10 15:44:14.931485 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.931466 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" Sep 10 15:44:14.931533 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.931490 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96f20ea-2bbd-4fd6-b09b-7c167e990314" containerName="istio-proxy" Sep 10 15:44:14.931633 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.931621 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" containerName="istio-proxy" Sep 10 15:44:14.934192 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.934174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:14.943570 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:14.943548 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr"] Sep 10 15:44:15.002177 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.002107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbb6g\" (UniqueName: \"kubernetes.io/projected/7bb53d64-8477-4b24-970f-141107d82e24-kube-api-access-mbb6g\") pod \"payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.002611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.002206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb53d64-8477-4b24-970f-141107d82e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.002611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.002266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.002611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.002342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbcr\" (UniqueName: \"kubernetes.io/projected/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-kube-api-access-8mbcr\") pod \"payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.103500 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.103461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbb6g\" (UniqueName: \"kubernetes.io/projected/7bb53d64-8477-4b24-970f-141107d82e24-kube-api-access-mbb6g\") pod \"payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.103718 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.103529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb53d64-8477-4b24-970f-141107d82e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.103718 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.103562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.103718 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.103608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbcr\" (UniqueName: \"kubernetes.io/projected/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-kube-api-access-8mbcr\") pod \"payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.104322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.104301 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.104383 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.104299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb53d64-8477-4b24-970f-141107d82e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.112906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.112876 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbb6g\" (UniqueName: \"kubernetes.io/projected/7bb53d64-8477-4b24-970f-141107d82e24-kube-api-access-mbb6g\") pod \"payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.113029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.112946 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbcr\" (UniqueName: \"kubernetes.io/projected/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-kube-api-access-8mbcr\") pod \"payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.133305 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.133276 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7e892e7-70b5-452a-b692-9c69da28944d" containerID="863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9" exitCode=0 Sep 10 15:44:15.133414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.133341 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" Sep 10 15:44:15.133414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.133358 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" event={"ID":"e7e892e7-70b5-452a-b692-9c69da28944d","Type":"ContainerDied","Data":"863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9"} Sep 10 15:44:15.133491 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.133417 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp" event={"ID":"e7e892e7-70b5-452a-b692-9c69da28944d","Type":"ContainerDied","Data":"9b190cd0ae597268bcbd9a80e04d085cebe36aefb6053abf80bc93f721e06660"} Sep 10 15:44:15.133491 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.133449 2581 scope.go:117] "RemoveContainer" containerID="863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9" Sep 10 15:44:15.142551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.142534 2581 scope.go:117] "RemoveContainer" containerID="3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4" Sep 10 15:44:15.153425 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.153406 2581 scope.go:117] "RemoveContainer" containerID="863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9" Sep 10 15:44:15.153755 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:44:15.153733 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9\": container with ID starting with 863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9 not found: ID does not exist" containerID="863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9" Sep 10 15:44:15.153823 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.153762 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9"} err="failed to get container status \"863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9\": rpc error: code = NotFound desc = could not find container \"863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9\": container with ID starting with 863b14283de3cb80d7f72bc43f53c3bcc0807ae4c4d812ed8af8fe3566367bd9 not found: ID does not exist" Sep 10 15:44:15.153823 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.153780 2581 scope.go:117] "RemoveContainer" containerID="3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4" Sep 10 15:44:15.154091 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:44:15.154073 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4\": container with ID starting with 3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4 not found: ID does not exist" containerID="3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4" Sep 10 15:44:15.154150 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.154097 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4"} err="failed to get container status \"3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4\": rpc error: code = NotFound desc = could not find container \"3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4\": container with ID starting with 3d253eae0864a49ba9907d4d0c611b780236aad924cf153462d060e4b8a6dfb4 not found: ID does not exist" Sep 10 15:44:15.154668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.154650 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp"] Sep 10 15:44:15.158941 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.158915 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-680721f9-openshift-default-54b498fb9b-hhwpp"] Sep 10 15:44:15.212937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.212903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:15.245379 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.245344 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:15.488234 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.488207 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg"] Sep 10 15:44:15.489487 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:44:15.489461 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb53d64_8477_4b24_970f_141107d82e24.slice/crio-a18a34c0c1f3911f2b07ada2815a269e48f3d37ea2694407486700330aea4141 WatchSource:0}: Error finding container a18a34c0c1f3911f2b07ada2815a269e48f3d37ea2694407486700330aea4141: Status 404 returned error can't find the container with id a18a34c0c1f3911f2b07ada2815a269e48f3d37ea2694407486700330aea4141 Sep 10 15:44:15.529369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:15.529343 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr"] Sep 10 15:44:15.534987 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:44:15.534963 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c5018b_717d_4db5_a2f4_f4f527ce0efa.slice/crio-7bad862f84ea5f0e63c5f25de238e02be5d98af28e699cbe1e776f312c5a86a0 WatchSource:0}: Error finding container 7bad862f84ea5f0e63c5f25de238e02be5d98af28e699cbe1e776f312c5a86a0: Status 404 returned error can't find the container with id 7bad862f84ea5f0e63c5f25de238e02be5d98af28e699cbe1e776f312c5a86a0 Sep 10 15:44:16.140711 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.140676 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" event={"ID":"c6c5018b-717d-4db5-a2f4-f4f527ce0efa","Type":"ContainerStarted","Data":"6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0"} Sep 10 15:44:16.140912 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.140737 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" event={"ID":"c6c5018b-717d-4db5-a2f4-f4f527ce0efa","Type":"ContainerStarted","Data":"7bad862f84ea5f0e63c5f25de238e02be5d98af28e699cbe1e776f312c5a86a0"} Sep 10 15:44:16.140912 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.140768 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:16.143009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.142969 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" event={"ID":"7bb53d64-8477-4b24-970f-141107d82e24","Type":"ContainerStarted","Data":"eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682"} Sep 10 15:44:16.143108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.143040 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" event={"ID":"7bb53d64-8477-4b24-970f-141107d82e24","Type":"ContainerStarted","Data":"a18a34c0c1f3911f2b07ada2815a269e48f3d37ea2694407486700330aea4141"} Sep 10 15:44:16.143108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.143071 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:44:16.157986 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.157930 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" podStartSLOduration=2.157912389 podStartE2EDuration="2.157912389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:16.157205029 +0000 UTC m=+1025.940418115" watchObservedRunningTime="2026-09-10 15:44:16.157912389 +0000 UTC m=+1025.941125454" Sep 10 15:44:16.175841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.175794 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" podStartSLOduration=2.175781315 podStartE2EDuration="2.175781315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:16.175129426 +0000 UTC m=+1025.958342492" watchObservedRunningTime="2026-09-10 15:44:16.175781315 +0000 UTC m=+1025.958994379" Sep 10 15:44:16.794199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:16.791857 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e892e7-70b5-452a-b692-9c69da28944d" path="/var/lib/kubelet/pods/e7e892e7-70b5-452a-b692-9c69da28944d/volumes" Sep 10 15:44:27.151168 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:27.151130 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:44:27.151624 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:44:27.151333 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:45:00.144919 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.144882 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97"] Sep 10 15:45:00.148922 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.148899 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8"] Sep 10 15:45:00.149083 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.149048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" Sep 10 15:45:00.151735 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.151716 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:45:00.152294 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.152273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" Sep 10 15:45:00.158601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.158578 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97"] Sep 10 15:45:00.161197 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.161176 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8"] Sep 10 15:45:00.163098 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.163078 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" Sep 10 15:45:00.171906 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.171738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" Sep 10 15:45:00.449258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.449213 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97"] Sep 10 15:45:00.451693 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:45:00.451669 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6663754c_5525_4ae7_994f_240d5e890bf3.slice/crio-ba9201abfa4b0d954a3038f587e856157ed813aee6543690e6ba2adf1caa6400 WatchSource:0}: Error finding container ba9201abfa4b0d954a3038f587e856157ed813aee6543690e6ba2adf1caa6400: Status 404 returned error can't find the container with id ba9201abfa4b0d954a3038f587e856157ed813aee6543690e6ba2adf1caa6400 Sep 10 15:45:00.452220 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:00.452197 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8"] Sep 10 15:45:00.453421 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:45:00.453402 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb7a433_3ea8_4b7f_a8b6_03c8ab1422f5.slice/crio-95f4ef9906a4b2cae2f27969c8f27e7d5121ef927493b7853f2d817f144520c2 WatchSource:0}: Error finding container 95f4ef9906a4b2cae2f27969c8f27e7d5121ef927493b7853f2d817f144520c2: Status 404 returned error can't find the container with id 95f4ef9906a4b2cae2f27969c8f27e7d5121ef927493b7853f2d817f144520c2 Sep 10 15:45:01.407349 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:01.407311 2581 generic.go:356] "Generic (PLEG): container finished" podID="2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5" containerID="4a3257b12cb8b4719f3cd7b530ab22574b29e996dce58c84dc004670317381bf" exitCode=0 Sep 10 15:45:01.407724 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:01.407387 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" event={"ID":"2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5","Type":"ContainerDied","Data":"4a3257b12cb8b4719f3cd7b530ab22574b29e996dce58c84dc004670317381bf"} Sep 10 15:45:01.407724 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:01.407442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" event={"ID":"2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5","Type":"ContainerStarted","Data":"95f4ef9906a4b2cae2f27969c8f27e7d5121ef927493b7853f2d817f144520c2"} Sep 10 15:45:01.409058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:01.409030 2581 generic.go:356] "Generic (PLEG): container finished" podID="6663754c-5525-4ae7-994f-240d5e890bf3" containerID="63f64763f93b7196b0ab98c8674fbdaf9d5c9b0449daacb7cd2f9ca54a66feed" exitCode=0 Sep 10 15:45:01.409196 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:01.409108 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" event={"ID":"6663754c-5525-4ae7-994f-240d5e890bf3","Type":"ContainerDied","Data":"63f64763f93b7196b0ab98c8674fbdaf9d5c9b0449daacb7cd2f9ca54a66feed"} Sep 10 15:45:01.409196 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:01.409164 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" event={"ID":"6663754c-5525-4ae7-994f-240d5e890bf3","Type":"ContainerStarted","Data":"ba9201abfa4b0d954a3038f587e856157ed813aee6543690e6ba2adf1caa6400"} Sep 10 15:45:02.568408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:02.568383 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" Sep 10 15:45:02.571384 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:02.571366 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" Sep 10 15:45:03.421225 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:03.421200 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" Sep 10 15:45:03.421430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:03.421249 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97" event={"ID":"6663754c-5525-4ae7-994f-240d5e890bf3","Type":"ContainerDied","Data":"ba9201abfa4b0d954a3038f587e856157ed813aee6543690e6ba2adf1caa6400"} Sep 10 15:45:03.421430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:03.421302 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9201abfa4b0d954a3038f587e856157ed813aee6543690e6ba2adf1caa6400" Sep 10 15:45:03.422728 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:03.422708 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" event={"ID":"2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5","Type":"ContainerDied","Data":"95f4ef9906a4b2cae2f27969c8f27e7d5121ef927493b7853f2d817f144520c2"} Sep 10 15:45:03.422825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:03.422732 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8" Sep 10 15:45:03.422825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:03.422742 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95f4ef9906a4b2cae2f27969c8f27e7d5121ef927493b7853f2d817f144520c2" Sep 10 15:45:09.049522 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.049483 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp"] Sep 10 15:45:09.050748 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.050714 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5" containerName="cleanup" Sep 10 15:45:09.050898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.050887 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5" containerName="cleanup" Sep 10 15:45:09.051029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.051017 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6663754c-5525-4ae7-994f-240d5e890bf3" containerName="cleanup" Sep 10 15:45:09.051136 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.051126 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6663754c-5525-4ae7-994f-240d5e890bf3" containerName="cleanup" Sep 10 15:45:09.051347 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.051333 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6663754c-5525-4ae7-994f-240d5e890bf3" containerName="cleanup" Sep 10 15:45:09.051435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.051426 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5" containerName="cleanup" Sep 10 15:45:09.055237 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.055212 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:09.059519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.059114 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:45:09.059519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.059376 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:45:09.061054 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.061034 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp"] Sep 10 15:45:09.148468 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.148427 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bffe16-9623-424d-87aa-1f28a1f3fac9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp\" (UID: \"f7bffe16-9623-424d-87aa-1f28a1f3fac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:09.249149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.249103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bffe16-9623-424d-87aa-1f28a1f3fac9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp\" (UID: \"f7bffe16-9623-424d-87aa-1f28a1f3fac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:09.249725 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.249702 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bffe16-9623-424d-87aa-1f28a1f3fac9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp\" (UID: \"f7bffe16-9623-424d-87aa-1f28a1f3fac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:09.372382 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.372295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:09.645670 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:09.645645 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp"] Sep 10 15:45:09.648187 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:45:09.648150 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bffe16_9623_424d_87aa_1f28a1f3fac9.slice/crio-92fa201474d67afee24eb773e17e789fef71187a62427747b87c0c605113e944 WatchSource:0}: Error finding container 92fa201474d67afee24eb773e17e789fef71187a62427747b87c0c605113e944: Status 404 returned error can't find the container with id 92fa201474d67afee24eb773e17e789fef71187a62427747b87c0c605113e944 Sep 10 15:45:10.465056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:10.465021 2581 generic.go:356] "Generic (PLEG): container finished" podID="f7bffe16-9623-424d-87aa-1f28a1f3fac9" containerID="0add825e498da10ee7726d854fc191472d0cb26f576abb7e8e99e59b5873db52" exitCode=0 Sep 10 15:45:10.465399 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:10.465121 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" event={"ID":"f7bffe16-9623-424d-87aa-1f28a1f3fac9","Type":"ContainerDied","Data":"0add825e498da10ee7726d854fc191472d0cb26f576abb7e8e99e59b5873db52"} Sep 10 15:45:10.465399 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:10.465171 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" event={"ID":"f7bffe16-9623-424d-87aa-1f28a1f3fac9","Type":"ContainerStarted","Data":"92fa201474d67afee24eb773e17e789fef71187a62427747b87c0c605113e944"} Sep 10 15:45:11.613825 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:11.613796 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:11.677484 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:11.677449 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bffe16-9623-424d-87aa-1f28a1f3fac9-maas-api-service-ca\") pod \"f7bffe16-9623-424d-87aa-1f28a1f3fac9\" (UID: \"f7bffe16-9623-424d-87aa-1f28a1f3fac9\") " Sep 10 15:45:11.677819 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:11.677792 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bffe16-9623-424d-87aa-1f28a1f3fac9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7bffe16-9623-424d-87aa-1f28a1f3fac9" (UID: "f7bffe16-9623-424d-87aa-1f28a1f3fac9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:11.778977 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:11.778875 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bffe16-9623-424d-87aa-1f28a1f3fac9-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:45:12.477845 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:12.477806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" event={"ID":"f7bffe16-9623-424d-87aa-1f28a1f3fac9","Type":"ContainerDied","Data":"92fa201474d67afee24eb773e17e789fef71187a62427747b87c0c605113e944"} Sep 10 15:45:12.477845 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:12.477845 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp" Sep 10 15:45:12.478096 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:12.477869 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fa201474d67afee24eb773e17e789fef71187a62427747b87c0c605113e944" Sep 10 15:45:19.721042 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.720970 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8"] Sep 10 15:45:19.723897 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.723873 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-fc495e-29817585-qctb8"] Sep 10 15:45:19.755307 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.755269 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr"] Sep 10 15:45:19.755551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.755521 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" podUID="c6c5018b-717d-4db5-a2f4-f4f527ce0efa" containerName="payload-processing" containerID="cri-o://6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0" gracePeriod=30 Sep 10 15:45:19.760890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.760863 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg"] Sep 10 15:45:19.761265 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.761235 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" podUID="7bb53d64-8477-4b24-970f-141107d82e24" containerName="payload-pre-processing" containerID="cri-o://eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682" gracePeriod=30 Sep 10 15:45:19.939330 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.939299 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp"] Sep 10 15:45:19.942939 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:19.942912 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fc495e-4qwgp"] Sep 10 15:45:20.056399 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.056370 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:45:20.058430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.058412 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:45:20.168501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.168451 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mbcr\" (UniqueName: \"kubernetes.io/projected/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-kube-api-access-8mbcr\") pod \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " Sep 10 15:45:20.168707 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.168579 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-plugins-config-volume\") pod \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\" (UID: \"c6c5018b-717d-4db5-a2f4-f4f527ce0efa\") " Sep 10 15:45:20.168707 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.168634 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbb6g\" (UniqueName: \"kubernetes.io/projected/7bb53d64-8477-4b24-970f-141107d82e24-kube-api-access-mbb6g\") pod \"7bb53d64-8477-4b24-970f-141107d82e24\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " Sep 10 15:45:20.168707 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.168653 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb53d64-8477-4b24-970f-141107d82e24-plugins-config-volume\") pod \"7bb53d64-8477-4b24-970f-141107d82e24\" (UID: \"7bb53d64-8477-4b24-970f-141107d82e24\") " Sep 10 15:45:20.169067 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.169041 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6c5018b-717d-4db5-a2f4-f4f527ce0efa" (UID: "c6c5018b-717d-4db5-a2f4-f4f527ce0efa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:20.169151 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.169073 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb53d64-8477-4b24-970f-141107d82e24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bb53d64-8477-4b24-970f-141107d82e24" (UID: "7bb53d64-8477-4b24-970f-141107d82e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:20.170605 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.170584 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-kube-api-access-8mbcr" (OuterVolumeSpecName: "kube-api-access-8mbcr") pod "c6c5018b-717d-4db5-a2f4-f4f527ce0efa" (UID: "c6c5018b-717d-4db5-a2f4-f4f527ce0efa"). InnerVolumeSpecName "kube-api-access-8mbcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:45:20.170605 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.170596 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb53d64-8477-4b24-970f-141107d82e24-kube-api-access-mbb6g" (OuterVolumeSpecName: "kube-api-access-mbb6g") pod "7bb53d64-8477-4b24-970f-141107d82e24" (UID: "7bb53d64-8477-4b24-970f-141107d82e24"). InnerVolumeSpecName "kube-api-access-mbb6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:45:20.269622 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.269525 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:45:20.269622 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.269581 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbb6g\" (UniqueName: \"kubernetes.io/projected/7bb53d64-8477-4b24-970f-141107d82e24-kube-api-access-mbb6g\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:45:20.269622 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.269593 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb53d64-8477-4b24-970f-141107d82e24-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:45:20.269622 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.269602 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mbcr\" (UniqueName: \"kubernetes.io/projected/c6c5018b-717d-4db5-a2f4-f4f527ce0efa-kube-api-access-8mbcr\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:45:20.524715 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.524632 2581 generic.go:356] "Generic (PLEG): container finished" podID="c6c5018b-717d-4db5-a2f4-f4f527ce0efa" containerID="6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0" exitCode=0 Sep 10 15:45:20.524715 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.524678 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" Sep 10 15:45:20.524925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.524718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" event={"ID":"c6c5018b-717d-4db5-a2f4-f4f527ce0efa","Type":"ContainerDied","Data":"6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0"} Sep 10 15:45:20.524925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.524767 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr" event={"ID":"c6c5018b-717d-4db5-a2f4-f4f527ce0efa","Type":"ContainerDied","Data":"7bad862f84ea5f0e63c5f25de238e02be5d98af28e699cbe1e776f312c5a86a0"} Sep 10 15:45:20.524925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.524789 2581 scope.go:117] "RemoveContainer" containerID="6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0" Sep 10 15:45:20.526105 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.526051 2581 generic.go:356] "Generic (PLEG): container finished" podID="7bb53d64-8477-4b24-970f-141107d82e24" containerID="eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682" exitCode=0 Sep 10 15:45:20.526189 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.526100 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" Sep 10 15:45:20.526189 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.526130 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" event={"ID":"7bb53d64-8477-4b24-970f-141107d82e24","Type":"ContainerDied","Data":"eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682"} Sep 10 15:45:20.526189 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.526166 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg" event={"ID":"7bb53d64-8477-4b24-970f-141107d82e24","Type":"ContainerDied","Data":"a18a34c0c1f3911f2b07ada2815a269e48f3d37ea2694407486700330aea4141"} Sep 10 15:45:20.535329 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.535309 2581 scope.go:117] "RemoveContainer" containerID="6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0" Sep 10 15:45:20.535600 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:45:20.535583 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0\": container with ID starting with 6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0 not found: ID does not exist" containerID="6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0" Sep 10 15:45:20.535679 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.535613 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0"} err="failed to get container status \"6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0\": rpc error: code = NotFound desc = could not find container \"6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0\": container with ID starting with 6e236a3e46e1dd08cc6501e244b82e9eb45a147e770ea4dfcc1e0dc6e43db5e0 not found: ID does not exist" Sep 10 15:45:20.535679 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.535636 2581 scope.go:117] "RemoveContainer" containerID="eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682" Sep 10 15:45:20.545849 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.545831 2581 scope.go:117] "RemoveContainer" containerID="eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682" Sep 10 15:45:20.546150 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:45:20.546130 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682\": container with ID starting with eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682 not found: ID does not exist" containerID="eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682" Sep 10 15:45:20.546234 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.546154 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682"} err="failed to get container status \"eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682\": rpc error: code = NotFound desc = could not find container \"eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682\": container with ID starting with eeb5e9188af117f2b7f946bfb53cce974aa2e33c9696f712bee6c9a71fdd5682 not found: ID does not exist" Sep 10 15:45:20.554778 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.554757 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg"] Sep 10 15:45:20.559436 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.559416 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fc495e-5987689874-2grrg"] Sep 10 15:45:20.570634 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.570610 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr"] Sep 10 15:45:20.575420 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.575388 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fc495e-bf7fc7b6f-m45hr"] Sep 10 15:45:20.793979 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.793953 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5" path="/var/lib/kubelet/pods/2fb7a433-3ea8-4b7f-a8b6-03c8ab1422f5/volumes" Sep 10 15:45:20.794310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.794296 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb53d64-8477-4b24-970f-141107d82e24" path="/var/lib/kubelet/pods/7bb53d64-8477-4b24-970f-141107d82e24/volumes" Sep 10 15:45:20.794611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.794598 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c5018b-717d-4db5-a2f4-f4f527ce0efa" path="/var/lib/kubelet/pods/c6c5018b-717d-4db5-a2f4-f4f527ce0efa/volumes" Sep 10 15:45:20.794884 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:20.794873 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bffe16-9623-424d-87aa-1f28a1f3fac9" path="/var/lib/kubelet/pods/f7bffe16-9623-424d-87aa-1f28a1f3fac9/volumes" Sep 10 15:45:27.901441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901408 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd"] Sep 10 15:45:27.901851 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901837 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb53d64-8477-4b24-970f-141107d82e24" containerName="payload-pre-processing" Sep 10 15:45:27.901901 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901853 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb53d64-8477-4b24-970f-141107d82e24" containerName="payload-pre-processing" Sep 10 15:45:27.901901 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901864 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5018b-717d-4db5-a2f4-f4f527ce0efa" containerName="payload-processing" Sep 10 15:45:27.901901 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901869 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5018b-717d-4db5-a2f4-f4f527ce0efa" containerName="payload-processing" Sep 10 15:45:27.901901 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901898 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bffe16-9623-424d-87aa-1f28a1f3fac9" containerName="revoke-keys" Sep 10 15:45:27.902092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901904 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bffe16-9623-424d-87aa-1f28a1f3fac9" containerName="revoke-keys" Sep 10 15:45:27.902092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901968 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c5018b-717d-4db5-a2f4-f4f527ce0efa" containerName="payload-processing" Sep 10 15:45:27.902092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901978 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb53d64-8477-4b24-970f-141107d82e24" containerName="payload-pre-processing" Sep 10 15:45:27.902092 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.901985 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bffe16-9623-424d-87aa-1f28a1f3fac9" containerName="revoke-keys" Sep 10 15:45:27.908224 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.908198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:27.911320 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.911295 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-2a38a4-openshift-default-dockercfg-9ng77\"" Sep 10 15:45:27.918041 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:27.917990 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd"] Sep 10 15:45:28.048707 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-credential-socket\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.048707 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7mr\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-kube-api-access-wr7mr\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.048927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048741 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-data\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.048927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88cb88d-6849-4241-b293-7fe1e1f3989b-istiod-ca-cert\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.048927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-podinfo\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.049052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-socket\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.049052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.048963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-envoy\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.049052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.049017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-certs\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.049176 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.049062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-token\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150217 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150176 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88cb88d-6849-4241-b293-7fe1e1f3989b-istiod-ca-cert\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-podinfo\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-socket\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-envoy\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-certs\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-token\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-credential-socket\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7mr\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-kube-api-access-wr7mr\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-data\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150752 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-socket\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150789 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-certs\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.150907 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-data\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.151109 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150960 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-credential-socket\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.151109 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.150981 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88cb88d-6849-4241-b293-7fe1e1f3989b-istiod-ca-cert\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.152787 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.152731 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-envoy\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.152918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.152899 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-podinfo\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.159242 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.159221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-token\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.159589 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.159569 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7mr\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-kube-api-access-wr7mr\") pod \"e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.222370 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.222331 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:28.501712 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.501685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd"] Sep 10 15:45:28.503357 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:45:28.503325 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88cb88d_6849_4241_b293_7fe1e1f3989b.slice/crio-e355efeb2dcdc0bf2480ecfddedf0a78e97071944d903fb4d4450708f430c759 WatchSource:0}: Error finding container e355efeb2dcdc0bf2480ecfddedf0a78e97071944d903fb4d4450708f430c759: Status 404 returned error can't find the container with id e355efeb2dcdc0bf2480ecfddedf0a78e97071944d903fb4d4450708f430c759 Sep 10 15:45:28.505556 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.505524 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:45:28.505668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.505605 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:45:28.505668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.505649 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:45:28.577026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.576985 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" event={"ID":"d88cb88d-6849-4241-b293-7fe1e1f3989b","Type":"ContainerStarted","Data":"d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214"} Sep 10 15:45:28.577154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.577049 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" event={"ID":"d88cb88d-6849-4241-b293-7fe1e1f3989b","Type":"ContainerStarted","Data":"e355efeb2dcdc0bf2480ecfddedf0a78e97071944d903fb4d4450708f430c759"} Sep 10 15:45:28.598023 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:28.597945 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" podStartSLOduration=1.597924031 podStartE2EDuration="1.597924031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:28.59558818 +0000 UTC m=+1098.378801245" watchObservedRunningTime="2026-09-10 15:45:28.597924031 +0000 UTC m=+1098.381137097" Sep 10 15:45:29.222559 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:29.222508 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:29.230398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:29.230369 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:29.583764 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:29.583728 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:29.584874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:29.584850 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:45:53.642198 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.642166 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg"] Sep 10 15:45:53.646314 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.646289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.649219 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.649195 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f4a9663c-dockercfg-mxtwk\"" Sep 10 15:45:53.649219 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.649205 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f4a9663c\"" Sep 10 15:45:53.662776 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.662749 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg"] Sep 10 15:45:53.717055 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.717023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2be257c-5b96-431f-8eb6-13a73babc978-plugins-config-volume\") pod \"payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.717301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.717176 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfw9c\" (UniqueName: \"kubernetes.io/projected/d2be257c-5b96-431f-8eb6-13a73babc978-kube-api-access-rfw9c\") pod \"payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.818526 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.818492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2be257c-5b96-431f-8eb6-13a73babc978-plugins-config-volume\") pod \"payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.818718 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.818614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfw9c\" (UniqueName: \"kubernetes.io/projected/d2be257c-5b96-431f-8eb6-13a73babc978-kube-api-access-rfw9c\") pod \"payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.819211 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.819193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2be257c-5b96-431f-8eb6-13a73babc978-plugins-config-volume\") pod \"payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.827892 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.827868 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfw9c\" (UniqueName: \"kubernetes.io/projected/d2be257c-5b96-431f-8eb6-13a73babc978-kube-api-access-rfw9c\") pod \"payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:53.957388 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:53.957303 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:54.229114 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:54.229086 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg"] Sep 10 15:45:54.230663 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:45:54.230636 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2be257c_5b96_431f_8eb6_13a73babc978.slice/crio-008f44c57e62eca87c982f695cccc8c2ee406b1eec9478e1de2b66420875f0f3 WatchSource:0}: Error finding container 008f44c57e62eca87c982f695cccc8c2ee406b1eec9478e1de2b66420875f0f3: Status 404 returned error can't find the container with id 008f44c57e62eca87c982f695cccc8c2ee406b1eec9478e1de2b66420875f0f3 Sep 10 15:45:54.726127 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:54.726065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" event={"ID":"d2be257c-5b96-431f-8eb6-13a73babc978","Type":"ContainerStarted","Data":"284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390"} Sep 10 15:45:54.726576 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:54.726133 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" event={"ID":"d2be257c-5b96-431f-8eb6-13a73babc978","Type":"ContainerStarted","Data":"008f44c57e62eca87c982f695cccc8c2ee406b1eec9478e1de2b66420875f0f3"} Sep 10 15:45:54.726576 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:54.726165 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:45:54.745891 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:45:54.745833 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" podStartSLOduration=1.7458150319999999 podStartE2EDuration="1.745815032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:54.742485977 +0000 UTC m=+1124.525699039" watchObservedRunningTime="2026-09-10 15:45:54.745815032 +0000 UTC m=+1124.529028102" Sep 10 15:46:00.808757 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.808721 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75"] Sep 10 15:46:00.812512 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.812495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:00.814645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.814622 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:46:00.814792 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.814772 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:46:00.819512 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.819488 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75"] Sep 10 15:46:00.893773 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.893738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bce7ea-af97-4418-9277-f11897e88320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-139155-jnb75\" (UID: \"d3bce7ea-af97-4418-9277-f11897e88320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:00.994662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.994629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bce7ea-af97-4418-9277-f11897e88320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-139155-jnb75\" (UID: \"d3bce7ea-af97-4418-9277-f11897e88320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:00.995288 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:00.995266 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bce7ea-af97-4418-9277-f11897e88320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-139155-jnb75\" (UID: \"d3bce7ea-af97-4418-9277-f11897e88320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:01.125426 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:01.125344 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:01.419029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:01.418216 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75"] Sep 10 15:46:01.424491 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:01.424460 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bce7ea_af97_4418_9277_f11897e88320.slice/crio-a3fd8dde0b2d0ee2f3fc86350a84fbd6d56c60cad232a65d2312e9af4b6dff3e WatchSource:0}: Error finding container a3fd8dde0b2d0ee2f3fc86350a84fbd6d56c60cad232a65d2312e9af4b6dff3e: Status 404 returned error can't find the container with id a3fd8dde0b2d0ee2f3fc86350a84fbd6d56c60cad232a65d2312e9af4b6dff3e Sep 10 15:46:01.769140 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:01.769109 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" event={"ID":"d3bce7ea-af97-4418-9277-f11897e88320","Type":"ContainerStarted","Data":"a3fd8dde0b2d0ee2f3fc86350a84fbd6d56c60cad232a65d2312e9af4b6dff3e"} Sep 10 15:46:02.777024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:02.776962 2581 generic.go:356] "Generic (PLEG): container finished" podID="d3bce7ea-af97-4418-9277-f11897e88320" containerID="39476d5a4a2eb914d58f32a9f4c62ad52b7f68b7aa82a930683122296370189a" exitCode=0 Sep 10 15:46:02.777417 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:02.777041 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" event={"ID":"d3bce7ea-af97-4418-9277-f11897e88320","Type":"ContainerDied","Data":"39476d5a4a2eb914d58f32a9f4c62ad52b7f68b7aa82a930683122296370189a"} Sep 10 15:46:03.919705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:03.919677 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:04.027029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.026964 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bce7ea-af97-4418-9277-f11897e88320-maas-api-service-ca\") pod \"d3bce7ea-af97-4418-9277-f11897e88320\" (UID: \"d3bce7ea-af97-4418-9277-f11897e88320\") " Sep 10 15:46:04.027334 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.027312 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bce7ea-af97-4418-9277-f11897e88320-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3bce7ea-af97-4418-9277-f11897e88320" (UID: "d3bce7ea-af97-4418-9277-f11897e88320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:04.128112 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.128077 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bce7ea-af97-4418-9277-f11897e88320-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:04.533197 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.533104 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd"] Sep 10 15:46:04.533361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.533328 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" podUID="d88cb88d-6849-4241-b293-7fe1e1f3989b" containerName="istio-proxy" containerID="cri-o://d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214" gracePeriod=30 Sep 10 15:46:04.791130 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.791054 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" Sep 10 15:46:04.791130 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.791067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75" event={"ID":"d3bce7ea-af97-4418-9277-f11897e88320","Type":"ContainerDied","Data":"a3fd8dde0b2d0ee2f3fc86350a84fbd6d56c60cad232a65d2312e9af4b6dff3e"} Sep 10 15:46:04.791130 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:04.791123 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3fd8dde0b2d0ee2f3fc86350a84fbd6d56c60cad232a65d2312e9af4b6dff3e" Sep 10 15:46:05.737465 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:05.737436 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:46:09.798580 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.798556 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:46:09.824575 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.824545 2581 generic.go:356] "Generic (PLEG): container finished" podID="d88cb88d-6849-4241-b293-7fe1e1f3989b" containerID="d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214" exitCode=0 Sep 10 15:46:09.824751 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.824592 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" Sep 10 15:46:09.824751 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.824627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" event={"ID":"d88cb88d-6849-4241-b293-7fe1e1f3989b","Type":"ContainerDied","Data":"d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214"} Sep 10 15:46:09.824751 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.824689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd" event={"ID":"d88cb88d-6849-4241-b293-7fe1e1f3989b","Type":"ContainerDied","Data":"e355efeb2dcdc0bf2480ecfddedf0a78e97071944d903fb4d4450708f430c759"} Sep 10 15:46:09.824751 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.824719 2581 scope.go:117] "RemoveContainer" containerID="d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214" Sep 10 15:46:09.834122 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.834097 2581 scope.go:117] "RemoveContainer" containerID="d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214" Sep 10 15:46:09.834403 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:46:09.834386 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214\": container with ID starting with d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214 not found: ID does not exist" containerID="d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214" Sep 10 15:46:09.834471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.834410 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214"} err="failed to get container status \"d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214\": rpc error: code = NotFound desc = could not find container \"d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214\": container with ID starting with d16b9921aa0585f79fe0a08e1f6c5b07fd8dd6dc5376903eb5a3ef69b7dce214 not found: ID does not exist" Sep 10 15:46:09.984209 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984172 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-data\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984209 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984213 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-socket\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984240 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-token\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984293 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88cb88d-6849-4241-b293-7fe1e1f3989b-istiod-ca-cert\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr7mr\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-kube-api-access-wr7mr\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984354 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-certs\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984386 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-credential-socket\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984506 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:09.984701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984516 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-envoy\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.984701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984589 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:09.984701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984617 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:09.984890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984693 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88cb88d-6849-4241-b293-7fe1e1f3989b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:09.984890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984694 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:09.984890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.984855 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-podinfo\") pod \"d88cb88d-6849-4241-b293-7fe1e1f3989b\" (UID: \"d88cb88d-6849-4241-b293-7fe1e1f3989b\") " Sep 10 15:46:09.985224 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.985208 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:09.985296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.985225 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:09.985296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.985240 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:09.985296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.985256 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88cb88d-6849-4241-b293-7fe1e1f3989b-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:09.985296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.985272 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:09.986647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.986621 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:09.986647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.986636 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-kube-api-access-wr7mr" (OuterVolumeSpecName: "kube-api-access-wr7mr") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "kube-api-access-wr7mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:09.986785 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.986643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:09.986858 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:09.986836 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d88cb88d-6849-4241-b293-7fe1e1f3989b" (UID: "d88cb88d-6849-4241-b293-7fe1e1f3989b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:46:10.086546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.086443 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:10.086546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.086479 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:10.086546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.086492 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7mr\" (UniqueName: \"kubernetes.io/projected/d88cb88d-6849-4241-b293-7fe1e1f3989b-kube-api-access-wr7mr\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:10.086546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.086505 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88cb88d-6849-4241-b293-7fe1e1f3989b-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:10.148908 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.148866 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd"] Sep 10 15:46:10.153091 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.153064 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-2a38a4-openshift-default-5b89dc86db-2mvnd"] Sep 10 15:46:10.788900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:10.788871 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88cb88d-6849-4241-b293-7fe1e1f3989b" path="/var/lib/kubelet/pods/d88cb88d-6849-4241-b293-7fe1e1f3989b/volumes" Sep 10 15:46:11.389724 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.389691 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97"] Sep 10 15:46:11.400837 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.396028 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-139155-29817585-ncf97"] Sep 10 15:46:11.433419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.433372 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b"] Sep 10 15:46:11.433678 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.433642 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" podUID="317eb4e4-5c0b-45cb-9397-f2d5183c1e95" containerName="payload-processing" containerID="cri-o://fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794" gracePeriod=30 Sep 10 15:46:11.628524 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.628487 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75"] Sep 10 15:46:11.630648 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.630614 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-139155-jnb75"] Sep 10 15:46:11.701920 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.701894 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:46:11.800059 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.800030 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-plugins-config-volume\") pod \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " Sep 10 15:46:11.800216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.800094 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtpsz\" (UniqueName: \"kubernetes.io/projected/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-kube-api-access-jtpsz\") pod \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\" (UID: \"317eb4e4-5c0b-45cb-9397-f2d5183c1e95\") " Sep 10 15:46:11.800387 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.800363 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "317eb4e4-5c0b-45cb-9397-f2d5183c1e95" (UID: "317eb4e4-5c0b-45cb-9397-f2d5183c1e95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:11.802188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.802158 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-kube-api-access-jtpsz" (OuterVolumeSpecName: "kube-api-access-jtpsz") pod "317eb4e4-5c0b-45cb-9397-f2d5183c1e95" (UID: "317eb4e4-5c0b-45cb-9397-f2d5183c1e95"). InnerVolumeSpecName "kube-api-access-jtpsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:11.837060 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.837024 2581 generic.go:356] "Generic (PLEG): container finished" podID="317eb4e4-5c0b-45cb-9397-f2d5183c1e95" containerID="fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794" exitCode=0 Sep 10 15:46:11.837214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.837067 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" Sep 10 15:46:11.837214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.837104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" event={"ID":"317eb4e4-5c0b-45cb-9397-f2d5183c1e95","Type":"ContainerDied","Data":"fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794"} Sep 10 15:46:11.837214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.837167 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b" event={"ID":"317eb4e4-5c0b-45cb-9397-f2d5183c1e95","Type":"ContainerDied","Data":"53ce619acde395adf7dfa71c7681319e1640a51602388897eaaf57e42e6045a0"} Sep 10 15:46:11.837214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.837196 2581 scope.go:117] "RemoveContainer" containerID="fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794" Sep 10 15:46:11.847273 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.847204 2581 scope.go:117] "RemoveContainer" containerID="fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794" Sep 10 15:46:11.847574 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:46:11.847553 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794\": container with ID starting with fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794 not found: ID does not exist" containerID="fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794" Sep 10 15:46:11.847654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.847584 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794"} err="failed to get container status \"fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794\": rpc error: code = NotFound desc = could not find container \"fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794\": container with ID starting with fdf35633586b070525d6d5ea4f43340c399add855dca6a6f77562bd9ecde8794 not found: ID does not exist" Sep 10 15:46:11.882114 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.882078 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b"] Sep 10 15:46:11.884011 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.883979 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-139155-7f54fd47bf-6977b"] Sep 10 15:46:11.901598 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.901573 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:11.901598 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:11.901594 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpsz\" (UniqueName: \"kubernetes.io/projected/317eb4e4-5c0b-45cb-9397-f2d5183c1e95-kube-api-access-jtpsz\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:12.788221 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:12.788188 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="317eb4e4-5c0b-45cb-9397-f2d5183c1e95" path="/var/lib/kubelet/pods/317eb4e4-5c0b-45cb-9397-f2d5183c1e95/volumes" Sep 10 15:46:12.788623 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:12.788526 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6663754c-5525-4ae7-994f-240d5e890bf3" path="/var/lib/kubelet/pods/6663754c-5525-4ae7-994f-240d5e890bf3/volumes" Sep 10 15:46:12.788780 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:12.788767 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bce7ea-af97-4418-9277-f11897e88320" path="/var/lib/kubelet/pods/d3bce7ea-af97-4418-9277-f11897e88320/volumes" Sep 10 15:46:14.378400 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378368 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47"] Sep 10 15:46:14.378801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378787 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="317eb4e4-5c0b-45cb-9397-f2d5183c1e95" containerName="payload-processing" Sep 10 15:46:14.378841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378802 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="317eb4e4-5c0b-45cb-9397-f2d5183c1e95" containerName="payload-processing" Sep 10 15:46:14.378841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378810 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88cb88d-6849-4241-b293-7fe1e1f3989b" containerName="istio-proxy" Sep 10 15:46:14.378841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378815 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88cb88d-6849-4241-b293-7fe1e1f3989b" containerName="istio-proxy" Sep 10 15:46:14.378841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378836 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bce7ea-af97-4418-9277-f11897e88320" containerName="revoke-keys" Sep 10 15:46:14.378841 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378841 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bce7ea-af97-4418-9277-f11897e88320" containerName="revoke-keys" Sep 10 15:46:14.379031 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378904 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="317eb4e4-5c0b-45cb-9397-f2d5183c1e95" containerName="payload-processing" Sep 10 15:46:14.379031 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378912 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88cb88d-6849-4241-b293-7fe1e1f3989b" containerName="istio-proxy" Sep 10 15:46:14.379031 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.378920 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bce7ea-af97-4418-9277-f11897e88320" containerName="revoke-keys" Sep 10 15:46:14.383407 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.383388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:14.386190 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.385978 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:46:14.386829 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.386805 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:46:14.390249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.390216 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47"] Sep 10 15:46:14.424241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.424208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fee41d-7c05-4ba9-87bb-83ee39671187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f4a9663c-cln47\" (UID: \"f9fee41d-7c05-4ba9-87bb-83ee39671187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:14.524801 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.524757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fee41d-7c05-4ba9-87bb-83ee39671187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f4a9663c-cln47\" (UID: \"f9fee41d-7c05-4ba9-87bb-83ee39671187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:14.525435 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.525413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fee41d-7c05-4ba9-87bb-83ee39671187-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f4a9663c-cln47\" (UID: \"f9fee41d-7c05-4ba9-87bb-83ee39671187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:14.696132 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.696052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:14.967777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:14.967751 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47"] Sep 10 15:46:14.969212 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:14.969183 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fee41d_7c05_4ba9_87bb_83ee39671187.slice/crio-46cf4aa6bdaa47f3eb406a37c86383414befa1bd2858ecf4135bf99a1ae19cc2 WatchSource:0}: Error finding container 46cf4aa6bdaa47f3eb406a37c86383414befa1bd2858ecf4135bf99a1ae19cc2: Status 404 returned error can't find the container with id 46cf4aa6bdaa47f3eb406a37c86383414befa1bd2858ecf4135bf99a1ae19cc2 Sep 10 15:46:15.862058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:15.862025 2581 generic.go:356] "Generic (PLEG): container finished" podID="f9fee41d-7c05-4ba9-87bb-83ee39671187" containerID="ac17f91fa2f70efd4d787c243bb819e0f862e1a2417dec94b4fd928c6396f61d" exitCode=0 Sep 10 15:46:15.862503 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:15.862111 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" event={"ID":"f9fee41d-7c05-4ba9-87bb-83ee39671187","Type":"ContainerDied","Data":"ac17f91fa2f70efd4d787c243bb819e0f862e1a2417dec94b4fd928c6396f61d"} Sep 10 15:46:15.862503 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:15.862157 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" event={"ID":"f9fee41d-7c05-4ba9-87bb-83ee39671187","Type":"ContainerStarted","Data":"46cf4aa6bdaa47f3eb406a37c86383414befa1bd2858ecf4135bf99a1ae19cc2"} Sep 10 15:46:17.001444 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.001417 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:17.051681 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.051633 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fee41d-7c05-4ba9-87bb-83ee39671187-maas-api-service-ca\") pod \"f9fee41d-7c05-4ba9-87bb-83ee39671187\" (UID: \"f9fee41d-7c05-4ba9-87bb-83ee39671187\") " Sep 10 15:46:17.052027 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.051983 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fee41d-7c05-4ba9-87bb-83ee39671187-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9fee41d-7c05-4ba9-87bb-83ee39671187" (UID: "f9fee41d-7c05-4ba9-87bb-83ee39671187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:17.153265 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.153169 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fee41d-7c05-4ba9-87bb-83ee39671187-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:17.875513 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.875481 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" Sep 10 15:46:17.875513 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.875488 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47" event={"ID":"f9fee41d-7c05-4ba9-87bb-83ee39671187","Type":"ContainerDied","Data":"46cf4aa6bdaa47f3eb406a37c86383414befa1bd2858ecf4135bf99a1ae19cc2"} Sep 10 15:46:17.875740 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:17.875538 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46cf4aa6bdaa47f3eb406a37c86383414befa1bd2858ecf4135bf99a1ae19cc2" Sep 10 15:46:18.682961 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:18.682928 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b"] Sep 10 15:46:18.683391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:18.683159 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" containerID="cri-o://edaeb35b84da853bbba754b27f90fa5e169274b587e8529927b4a85b4d4ab410" gracePeriod=30 Sep 10 15:46:19.058390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.058358 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc"] Sep 10 15:46:19.058982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.058955 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fee41d-7c05-4ba9-87bb-83ee39671187" containerName="revoke-keys" Sep 10 15:46:19.058982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.058982 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fee41d-7c05-4ba9-87bb-83ee39671187" containerName="revoke-keys" Sep 10 15:46:19.059193 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.059124 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fee41d-7c05-4ba9-87bb-83ee39671187" containerName="revoke-keys" Sep 10 15:46:19.062342 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.062321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:19.064585 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.064564 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:46:19.064676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.064651 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:46:19.070138 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.070080 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc"] Sep 10 15:46:19.173811 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.173782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/548bdcee-35ca-474a-957e-af0afea3991d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc\" (UID: \"548bdcee-35ca-474a-957e-af0afea3991d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:19.275271 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.275235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/548bdcee-35ca-474a-957e-af0afea3991d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc\" (UID: \"548bdcee-35ca-474a-957e-af0afea3991d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:19.275888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.275867 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/548bdcee-35ca-474a-957e-af0afea3991d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc\" (UID: \"548bdcee-35ca-474a-957e-af0afea3991d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:19.374803 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.374736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:19.646036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.646008 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc"] Sep 10 15:46:19.647144 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:19.647117 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548bdcee_35ca_474a_957e_af0afea3991d.slice/crio-cae9e402daab23806d5bf2b5e9d44d2204ece7da216479c696642f82bf273d7b WatchSource:0}: Error finding container cae9e402daab23806d5bf2b5e9d44d2204ece7da216479c696642f82bf273d7b: Status 404 returned error can't find the container with id cae9e402daab23806d5bf2b5e9d44d2204ece7da216479c696642f82bf273d7b Sep 10 15:46:19.888332 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.888305 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" event={"ID":"548bdcee-35ca-474a-957e-af0afea3991d","Type":"ContainerStarted","Data":"f4c0112f0519b7f87d948645a87fe0a0dfa13d377d05edc96a26e200139bc165"} Sep 10 15:46:19.888646 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.888354 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" event={"ID":"548bdcee-35ca-474a-957e-af0afea3991d","Type":"ContainerStarted","Data":"cae9e402daab23806d5bf2b5e9d44d2204ece7da216479c696642f82bf273d7b"} Sep 10 15:46:19.907237 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:19.907126 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" podStartSLOduration=0.907109277 podStartE2EDuration="907.109277ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:19.905215852 +0000 UTC m=+1149.688428917" watchObservedRunningTime="2026-09-10 15:46:19.907109277 +0000 UTC m=+1149.690322341" Sep 10 15:46:20.895873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:20.895837 2581 generic.go:356] "Generic (PLEG): container finished" podID="548bdcee-35ca-474a-957e-af0afea3991d" containerID="f4c0112f0519b7f87d948645a87fe0a0dfa13d377d05edc96a26e200139bc165" exitCode=0 Sep 10 15:46:20.896316 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:20.895930 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" event={"ID":"548bdcee-35ca-474a-957e-af0afea3991d","Type":"ContainerDied","Data":"f4c0112f0519b7f87d948645a87fe0a0dfa13d377d05edc96a26e200139bc165"} Sep 10 15:46:22.033747 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.033719 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:22.100900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.100865 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/548bdcee-35ca-474a-957e-af0afea3991d-maas-api-service-ca\") pod \"548bdcee-35ca-474a-957e-af0afea3991d\" (UID: \"548bdcee-35ca-474a-957e-af0afea3991d\") " Sep 10 15:46:22.101218 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.101196 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548bdcee-35ca-474a-957e-af0afea3991d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "548bdcee-35ca-474a-957e-af0afea3991d" (UID: "548bdcee-35ca-474a-957e-af0afea3991d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:22.202509 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.202428 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/548bdcee-35ca-474a-957e-af0afea3991d-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:22.908944 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.908910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" event={"ID":"548bdcee-35ca-474a-957e-af0afea3991d","Type":"ContainerDied","Data":"cae9e402daab23806d5bf2b5e9d44d2204ece7da216479c696642f82bf273d7b"} Sep 10 15:46:22.909160 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.908956 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae9e402daab23806d5bf2b5e9d44d2204ece7da216479c696642f82bf273d7b" Sep 10 15:46:22.909160 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:22.908956 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc" Sep 10 15:46:23.428067 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.428031 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg"] Sep 10 15:46:23.428890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.428832 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" podUID="d2be257c-5b96-431f-8eb6-13a73babc978" containerName="payload-processing" containerID="cri-o://284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390" gracePeriod=30 Sep 10 15:46:23.624376 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.624316 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47"] Sep 10 15:46:23.628518 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.628484 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f4a9663c-cln47"] Sep 10 15:46:23.680690 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.680625 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:46:23.716445 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.716384 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfw9c\" (UniqueName: \"kubernetes.io/projected/d2be257c-5b96-431f-8eb6-13a73babc978-kube-api-access-rfw9c\") pod \"d2be257c-5b96-431f-8eb6-13a73babc978\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " Sep 10 15:46:23.716628 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.716525 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2be257c-5b96-431f-8eb6-13a73babc978-plugins-config-volume\") pod \"d2be257c-5b96-431f-8eb6-13a73babc978\" (UID: \"d2be257c-5b96-431f-8eb6-13a73babc978\") " Sep 10 15:46:23.717217 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.717175 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2be257c-5b96-431f-8eb6-13a73babc978-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2be257c-5b96-431f-8eb6-13a73babc978" (UID: "d2be257c-5b96-431f-8eb6-13a73babc978"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:23.720502 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.720455 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2be257c-5b96-431f-8eb6-13a73babc978-kube-api-access-rfw9c" (OuterVolumeSpecName: "kube-api-access-rfw9c") pod "d2be257c-5b96-431f-8eb6-13a73babc978" (UID: "d2be257c-5b96-431f-8eb6-13a73babc978"). InnerVolumeSpecName "kube-api-access-rfw9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:23.818299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.818265 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2be257c-5b96-431f-8eb6-13a73babc978-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.818299 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.818301 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfw9c\" (UniqueName: \"kubernetes.io/projected/d2be257c-5b96-431f-8eb6-13a73babc978-kube-api-access-rfw9c\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:23.917278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.917248 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerID="edaeb35b84da853bbba754b27f90fa5e169274b587e8529927b4a85b4d4ab410" exitCode=0 Sep 10 15:46:23.917422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.917321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" event={"ID":"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236","Type":"ContainerDied","Data":"edaeb35b84da853bbba754b27f90fa5e169274b587e8529927b4a85b4d4ab410"} Sep 10 15:46:23.917422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.917378 2581 scope.go:117] "RemoveContainer" containerID="f0abe35e07a99ca5ac9c23db9a9d1b5c080a76d2ad25fd49af0f1cb3de95d401" Sep 10 15:46:23.918972 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.918950 2581 generic.go:356] "Generic (PLEG): container finished" podID="d2be257c-5b96-431f-8eb6-13a73babc978" containerID="284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390" exitCode=0 Sep 10 15:46:23.919135 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.919056 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" Sep 10 15:46:23.919135 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.919106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" event={"ID":"d2be257c-5b96-431f-8eb6-13a73babc978","Type":"ContainerDied","Data":"284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390"} Sep 10 15:46:23.919230 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.919153 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg" event={"ID":"d2be257c-5b96-431f-8eb6-13a73babc978","Type":"ContainerDied","Data":"008f44c57e62eca87c982f695cccc8c2ee406b1eec9478e1de2b66420875f0f3"} Sep 10 15:46:23.922927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.922904 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:46:23.926881 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.926714 2581 scope.go:117] "RemoveContainer" containerID="284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390" Sep 10 15:46:23.935337 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.935318 2581 scope.go:117] "RemoveContainer" containerID="284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390" Sep 10 15:46:23.935609 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:46:23.935590 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390\": container with ID starting with 284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390 not found: ID does not exist" containerID="284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390" Sep 10 15:46:23.935667 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.935616 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390"} err="failed to get container status \"284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390\": rpc error: code = NotFound desc = could not find container \"284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390\": container with ID starting with 284e97e313cc079ff4632796e4fc9858a3c76e5cdbb26c920231b1f6f6527390 not found: ID does not exist" Sep 10 15:46:23.961933 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.961901 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg"] Sep 10 15:46:23.964058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:23.964033 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4a9663c-6b766d5585-b57cg"] Sep 10 15:46:24.020041 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.019987 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-envoy\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020059 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-data\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020099 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-podinfo\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020150 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-certs\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020190 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-socket\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020207 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69ptn\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-kube-api-access-69ptn\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020262 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-token\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020261 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:24.020396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020285 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istiod-ca-cert\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020510 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-credential-socket\") pod \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\" (UID: \"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236\") " Sep 10 15:46:24.020602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020520 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:24.020602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020533 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:24.020728 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020606 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:24.020794 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.020775 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:24.021260 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.021174 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.021260 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.021200 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.021260 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.021215 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.021260 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.021227 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.021676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.021240 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.022579 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.022551 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:24.022774 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.022755 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:24.022875 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.022858 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-kube-api-access-69ptn" (OuterVolumeSpecName: "kube-api-access-69ptn") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "kube-api-access-69ptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:24.023006 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.022968 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" (UID: "e5bbfd2b-67dd-4fa6-90e1-e6d17db73236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:46:24.122861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.122824 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69ptn\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-kube-api-access-69ptn\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.122861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.122859 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.123137 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.122874 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.123137 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.122886 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.788321 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.788218 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2be257c-5b96-431f-8eb6-13a73babc978" path="/var/lib/kubelet/pods/d2be257c-5b96-431f-8eb6-13a73babc978/volumes" Sep 10 15:46:24.788790 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.788766 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fee41d-7c05-4ba9-87bb-83ee39671187" path="/var/lib/kubelet/pods/f9fee41d-7c05-4ba9-87bb-83ee39671187/volumes" Sep 10 15:46:24.925694 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.925659 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" Sep 10 15:46:24.925887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.925715 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b" event={"ID":"e5bbfd2b-67dd-4fa6-90e1-e6d17db73236","Type":"ContainerDied","Data":"8678a18a99aa32eaaf7116af2dd09fb57f24516ad7db7495d1a32c5362451c94"} Sep 10 15:46:24.925887 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.925780 2581 scope.go:117] "RemoveContainer" containerID="edaeb35b84da853bbba754b27f90fa5e169274b587e8529927b4a85b4d4ab410" Sep 10 15:46:24.978180 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.978138 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b"] Sep 10 15:46:24.982154 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:24.982126 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-139155-openshift-default-77d5bd8f46-snl4b"] Sep 10 15:46:26.789425 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:26.789396 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" path="/var/lib/kubelet/pods/e5bbfd2b-67dd-4fa6-90e1-e6d17db73236/volumes" Sep 10 15:46:29.789845 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:29.789810 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg"] Sep 10 15:46:29.790344 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:29.790044 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" podUID="e8e1a09c-0288-485f-8fd8-7df3b76367b7" containerName="payload-processing" containerID="cri-o://571ef218c6f0e79c14a659d4d6d0e56ce7aaa379894283fcb7d7b32b2ca6e82d" gracePeriod=30 Sep 10 15:46:29.958215 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:29.958183 2581 generic.go:356] "Generic (PLEG): container finished" podID="e8e1a09c-0288-485f-8fd8-7df3b76367b7" containerID="571ef218c6f0e79c14a659d4d6d0e56ce7aaa379894283fcb7d7b32b2ca6e82d" exitCode=0 Sep 10 15:46:29.958385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:29.958261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" event={"ID":"e8e1a09c-0288-485f-8fd8-7df3b76367b7","Type":"ContainerDied","Data":"571ef218c6f0e79c14a659d4d6d0e56ce7aaa379894283fcb7d7b32b2ca6e82d"} Sep 10 15:46:29.980141 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:29.980110 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc"] Sep 10 15:46:29.986577 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:29.986534 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90ca97-rrknc"] Sep 10 15:46:30.050963 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.050937 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:46:30.185163 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.185131 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdqxf\" (UniqueName: \"kubernetes.io/projected/e8e1a09c-0288-485f-8fd8-7df3b76367b7-kube-api-access-hdqxf\") pod \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " Sep 10 15:46:30.185163 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.185165 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1a09c-0288-485f-8fd8-7df3b76367b7-plugins-config-volume\") pod \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\" (UID: \"e8e1a09c-0288-485f-8fd8-7df3b76367b7\") " Sep 10 15:46:30.185608 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.185576 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e1a09c-0288-485f-8fd8-7df3b76367b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8e1a09c-0288-485f-8fd8-7df3b76367b7" (UID: "e8e1a09c-0288-485f-8fd8-7df3b76367b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:30.187283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.187259 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e1a09c-0288-485f-8fd8-7df3b76367b7-kube-api-access-hdqxf" (OuterVolumeSpecName: "kube-api-access-hdqxf") pod "e8e1a09c-0288-485f-8fd8-7df3b76367b7" (UID: "e8e1a09c-0288-485f-8fd8-7df3b76367b7"). InnerVolumeSpecName "kube-api-access-hdqxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:30.286554 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.286507 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqxf\" (UniqueName: \"kubernetes.io/projected/e8e1a09c-0288-485f-8fd8-7df3b76367b7-kube-api-access-hdqxf\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:30.286554 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.286544 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e1a09c-0288-485f-8fd8-7df3b76367b7-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:46:30.789356 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.789323 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548bdcee-35ca-474a-957e-af0afea3991d" path="/var/lib/kubelet/pods/548bdcee-35ca-474a-957e-af0afea3991d/volumes" Sep 10 15:46:30.964834 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.964809 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" Sep 10 15:46:30.964834 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.964818 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg" event={"ID":"e8e1a09c-0288-485f-8fd8-7df3b76367b7","Type":"ContainerDied","Data":"1b722e66966c2980be5373851fcf8171e76dc2c02f8c873a23f56605ba200296"} Sep 10 15:46:30.965379 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.964872 2581 scope.go:117] "RemoveContainer" containerID="571ef218c6f0e79c14a659d4d6d0e56ce7aaa379894283fcb7d7b32b2ca6e82d" Sep 10 15:46:30.995227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:30.995197 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg"] Sep 10 15:46:31.000702 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:31.000661 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90ca97-59f7ff644f-gsrqg"] Sep 10 15:46:32.789214 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:32.789179 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e1a09c-0288-485f-8fd8-7df3b76367b7" path="/var/lib/kubelet/pods/e8e1a09c-0288-485f-8fd8-7df3b76367b7/volumes" Sep 10 15:46:37.704969 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.704931 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4"] Sep 10 15:46:37.705590 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705568 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e1a09c-0288-485f-8fd8-7df3b76367b7" containerName="payload-processing" Sep 10 15:46:37.705644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705594 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e1a09c-0288-485f-8fd8-7df3b76367b7" containerName="payload-processing" Sep 10 15:46:37.705644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705607 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2be257c-5b96-431f-8eb6-13a73babc978" containerName="payload-processing" Sep 10 15:46:37.705644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705615 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2be257c-5b96-431f-8eb6-13a73babc978" containerName="payload-processing" Sep 10 15:46:37.705644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705626 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" Sep 10 15:46:37.705644 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705635 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" Sep 10 15:46:37.705800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705658 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" Sep 10 15:46:37.705800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705666 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" Sep 10 15:46:37.705800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705697 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548bdcee-35ca-474a-957e-af0afea3991d" containerName="revoke-keys" Sep 10 15:46:37.705800 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705704 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="548bdcee-35ca-474a-957e-af0afea3991d" containerName="revoke-keys" Sep 10 15:46:37.705925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705824 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" Sep 10 15:46:37.705925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705835 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e1a09c-0288-485f-8fd8-7df3b76367b7" containerName="payload-processing" Sep 10 15:46:37.705925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705847 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2be257c-5b96-431f-8eb6-13a73babc978" containerName="payload-processing" Sep 10 15:46:37.705925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.705858 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="548bdcee-35ca-474a-957e-af0afea3991d" containerName="revoke-keys" Sep 10 15:46:37.711934 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.711899 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.714898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.714870 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c7257b-openshift-default-dockercfg-2tb8n\"" Sep 10 15:46:37.719418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.719391 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4"] Sep 10 15:46:37.862562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.862531 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istiod-ca-cert\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.862732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.862586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-credential-socket\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.862732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.862639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-certs\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.862732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.862710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-envoy\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.862732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.862730 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-data\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.862955 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.862930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-token\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.863051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.863033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-podinfo\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.863103 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.863090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9sv\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-kube-api-access-6m9sv\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.863144 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.863134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-socket\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964537 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-credential-socket\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964537 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-certs\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964537 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-envoy\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-data\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-token\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-podinfo\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.964784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9sv\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-kube-api-access-6m9sv\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-socket\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istiod-ca-cert\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964937 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-certs\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.964966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-credential-socket\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965287 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.965049 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-data\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965287 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.965273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-socket\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.965501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.965476 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istiod-ca-cert\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.966618 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.966597 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-envoy\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.967263 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.967244 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-podinfo\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.978573 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.978540 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-token\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:37.978647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:37.978550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9sv\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-kube-api-access-6m9sv\") pod \"e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:38.025438 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:38.025399 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:38.306445 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:38.306417 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4"] Sep 10 15:46:38.308394 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:38.308359 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode377d128_6ae0_4fd3_bb68_3f8ad9f3cdbe.slice/crio-6444099d8f80d463405d38777a4f0ab032d7809e31dad9b6602495fdb58dd193 WatchSource:0}: Error finding container 6444099d8f80d463405d38777a4f0ab032d7809e31dad9b6602495fdb58dd193: Status 404 returned error can't find the container with id 6444099d8f80d463405d38777a4f0ab032d7809e31dad9b6602495fdb58dd193 Sep 10 15:46:38.310607 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:38.310577 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:46:38.310701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:38.310653 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:46:38.310701 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:38.310696 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:46:39.014373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:39.014339 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" event={"ID":"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe","Type":"ContainerStarted","Data":"7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52"} Sep 10 15:46:39.014734 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:39.014393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" event={"ID":"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe","Type":"ContainerStarted","Data":"6444099d8f80d463405d38777a4f0ab032d7809e31dad9b6602495fdb58dd193"} Sep 10 15:46:39.026458 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:39.026432 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:39.031918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:39.031894 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:39.043278 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:39.043214 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podStartSLOduration=2.043188561 podStartE2EDuration="2.043188561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:39.041895712 +0000 UTC m=+1168.825108779" watchObservedRunningTime="2026-09-10 15:46:39.043188561 +0000 UTC m=+1168.826401627" Sep 10 15:46:40.020060 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:40.020026 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:40.020955 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:40.020938 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:42.874224 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:42.874186 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2"] Sep 10 15:46:42.874659 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:42.874645 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5bbfd2b-67dd-4fa6-90e1-e6d17db73236" containerName="istio-proxy" Sep 10 15:46:42.877793 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:42.877778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:42.880029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:42.880008 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c7257b-dockercfg-f8fv8\"" Sep 10 15:46:42.880152 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:42.879991 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c7257b\"" Sep 10 15:46:42.887659 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:42.887634 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2"] Sep 10 15:46:43.016408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.016373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh2q\" (UniqueName: \"kubernetes.io/projected/4cd2033b-1347-4acb-9831-e5401337ce32-kube-api-access-wkh2q\") pod \"payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.016408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.016414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd2033b-1347-4acb-9831-e5401337ce32-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.117806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.117768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh2q\" (UniqueName: \"kubernetes.io/projected/4cd2033b-1347-4acb-9831-e5401337ce32-kube-api-access-wkh2q\") pod \"payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.117806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.117806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd2033b-1347-4acb-9831-e5401337ce32-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.118516 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.118495 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd2033b-1347-4acb-9831-e5401337ce32-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.127714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.127659 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh2q\" (UniqueName: \"kubernetes.io/projected/4cd2033b-1347-4acb-9831-e5401337ce32-kube-api-access-wkh2q\") pod \"payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.189143 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.189111 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:43.465911 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:43.465878 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2"] Sep 10 15:46:43.467538 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:43.467508 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd2033b_1347_4acb_9831_e5401337ce32.slice/crio-591f879734509a0e7cd067eb3fcccc5dba59ccfc9bb9014dcd17143604460c8b WatchSource:0}: Error finding container 591f879734509a0e7cd067eb3fcccc5dba59ccfc9bb9014dcd17143604460c8b: Status 404 returned error can't find the container with id 591f879734509a0e7cd067eb3fcccc5dba59ccfc9bb9014dcd17143604460c8b Sep 10 15:46:44.051971 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:44.051941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" event={"ID":"4cd2033b-1347-4acb-9831-e5401337ce32","Type":"ContainerStarted","Data":"8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba"} Sep 10 15:46:44.052469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:44.051988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" event={"ID":"4cd2033b-1347-4acb-9831-e5401337ce32","Type":"ContainerStarted","Data":"591f879734509a0e7cd067eb3fcccc5dba59ccfc9bb9014dcd17143604460c8b"} Sep 10 15:46:44.052469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:44.052037 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:44.068480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:44.068427 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" podStartSLOduration=2.068410911 podStartE2EDuration="2.068410911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:44.067472737 +0000 UTC m=+1173.850685802" watchObservedRunningTime="2026-09-10 15:46:44.068410911 +0000 UTC m=+1173.851623979" Sep 10 15:46:52.398474 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.398391 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m"] Sep 10 15:46:52.403401 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.403383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.405742 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.405720 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-30d70c\"" Sep 10 15:46:52.410247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.410221 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m"] Sep 10 15:46:52.415560 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.415529 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbde40a-c3f6-4b43-9f9a-12268f61831f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.415665 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.415601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7pvv\" (UniqueName: \"kubernetes.io/projected/8dbde40a-c3f6-4b43-9f9a-12268f61831f-kube-api-access-l7pvv\") pod \"payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.426452 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.426427 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn"] Sep 10 15:46:52.442779 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.442749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.443111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.443083 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn"] Sep 10 15:46:52.445108 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.445089 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-30d70c-dockercfg-f5cxg\"" Sep 10 15:46:52.516754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.516717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46a84a-bc32-4c70-a61e-295ef2a148b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.516922 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.516770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpw2\" (UniqueName: \"kubernetes.io/projected/3d46a84a-bc32-4c70-a61e-295ef2a148b9-kube-api-access-zlpw2\") pod \"payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.516922 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.516881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbde40a-c3f6-4b43-9f9a-12268f61831f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.517042 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.516931 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7pvv\" (UniqueName: \"kubernetes.io/projected/8dbde40a-c3f6-4b43-9f9a-12268f61831f-kube-api-access-l7pvv\") pod \"payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.517823 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.517796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbde40a-c3f6-4b43-9f9a-12268f61831f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.529138 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.528970 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7pvv\" (UniqueName: \"kubernetes.io/projected/8dbde40a-c3f6-4b43-9f9a-12268f61831f-kube-api-access-l7pvv\") pod \"payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.618047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.617976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46a84a-bc32-4c70-a61e-295ef2a148b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.618231 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.618069 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpw2\" (UniqueName: \"kubernetes.io/projected/3d46a84a-bc32-4c70-a61e-295ef2a148b9-kube-api-access-zlpw2\") pod \"payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.618767 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.618747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46a84a-bc32-4c70-a61e-295ef2a148b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.627662 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.627631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpw2\" (UniqueName: \"kubernetes.io/projected/3d46a84a-bc32-4c70-a61e-295ef2a148b9-kube-api-access-zlpw2\") pod \"payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.716440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.716339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:52.754837 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.754808 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:52.998247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:52.998219 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m"] Sep 10 15:46:53.000012 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:52.999966 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbde40a_c3f6_4b43_9f9a_12268f61831f.slice/crio-11f5674aca9c711ca19ded1f47258a248977cb267ad392bc32b92e7c63e0489e WatchSource:0}: Error finding container 11f5674aca9c711ca19ded1f47258a248977cb267ad392bc32b92e7c63e0489e: Status 404 returned error can't find the container with id 11f5674aca9c711ca19ded1f47258a248977cb267ad392bc32b92e7c63e0489e Sep 10 15:46:53.051035 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:53.051011 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn"] Sep 10 15:46:53.052625 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:46:53.052597 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d46a84a_bc32_4c70_a61e_295ef2a148b9.slice/crio-03be715ec6968ea1f97154171eeaa752855af5f4918a6a504936d64fff76ee80 WatchSource:0}: Error finding container 03be715ec6968ea1f97154171eeaa752855af5f4918a6a504936d64fff76ee80: Status 404 returned error can't find the container with id 03be715ec6968ea1f97154171eeaa752855af5f4918a6a504936d64fff76ee80 Sep 10 15:46:53.112227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:53.112199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" event={"ID":"8dbde40a-c3f6-4b43-9f9a-12268f61831f","Type":"ContainerStarted","Data":"ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32"} Sep 10 15:46:53.112372 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:53.112257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" event={"ID":"8dbde40a-c3f6-4b43-9f9a-12268f61831f","Type":"ContainerStarted","Data":"11f5674aca9c711ca19ded1f47258a248977cb267ad392bc32b92e7c63e0489e"} Sep 10 15:46:53.112372 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:53.112324 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:46:53.113959 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:53.113937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" event={"ID":"3d46a84a-bc32-4c70-a61e-295ef2a148b9","Type":"ContainerStarted","Data":"03be715ec6968ea1f97154171eeaa752855af5f4918a6a504936d64fff76ee80"} Sep 10 15:46:53.130362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:53.130300 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" podStartSLOduration=1.1302819259999999 podStartE2EDuration="1.130281926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:53.128114079 +0000 UTC m=+1182.911327147" watchObservedRunningTime="2026-09-10 15:46:53.130281926 +0000 UTC m=+1182.913494990" Sep 10 15:46:54.122814 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:54.122774 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" event={"ID":"3d46a84a-bc32-4c70-a61e-295ef2a148b9","Type":"ContainerStarted","Data":"21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962"} Sep 10 15:46:54.123256 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:54.123006 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:46:54.143485 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:54.143425 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" podStartSLOduration=2.143406109 podStartE2EDuration="2.143406109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:54.141349211 +0000 UTC m=+1183.924562276" watchObservedRunningTime="2026-09-10 15:46:54.143406109 +0000 UTC m=+1183.926619177" Sep 10 15:46:55.060440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:55.060408 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:46:55.203217 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:55.203183 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": read tcp 10.134.0.2:57666->10.134.0.65:15021: read: connection reset by peer" start-of-body= Sep 10 15:46:55.203818 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:55.203253 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": read tcp 10.134.0.2:57666->10.134.0.65:15021: read: connection reset by peer" Sep 10 15:46:56.137359 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:56.137326 2581 generic.go:356] "Generic (PLEG): container finished" podID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerID="7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52" exitCode=0 Sep 10 15:46:56.137528 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:56.137401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" event={"ID":"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe","Type":"ContainerDied","Data":"7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52"} Sep 10 15:46:56.137890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:56.137873 2581 scope.go:117] "RemoveContainer" containerID="7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52" Sep 10 15:46:56.138959 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:56.138911 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:46:56.139045 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:56.139015 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:46:56.139089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:56.139062 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:46:57.156093 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:57.156046 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" event={"ID":"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe","Type":"ContainerStarted","Data":"f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997"} Sep 10 15:46:58.026257 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:58.026219 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:46:58.026459 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:58.026439 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 15:46:58.026512 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:58.026484 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 15:46:59.026263 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:59.026227 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 15:46:59.026636 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:46:59.026285 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 15:47:00.026301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:00.026264 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 15:47:00.026682 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:00.026321 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 15:47:01.026051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:01.026008 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 15:47:01.026262 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:01.026081 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 15:47:02.026318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.026282 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 15:47:02.026700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.026343 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 15:47:02.565677 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.565644 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh"] Sep 10 15:47:02.568498 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.568481 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.571283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.571262 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3650b0-dockercfg-vjwcp\"" Sep 10 15:47:02.571406 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.571295 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3650b0\"" Sep 10 15:47:02.587976 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.587941 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh"] Sep 10 15:47:02.623518 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.623479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcrs\" (UniqueName: \"kubernetes.io/projected/c7d071f4-6d6e-4052-912e-901053f6d5bf-kube-api-access-9pcrs\") pod \"payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.623699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.623531 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d071f4-6d6e-4052-912e-901053f6d5bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.725187 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.725152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcrs\" (UniqueName: \"kubernetes.io/projected/c7d071f4-6d6e-4052-912e-901053f6d5bf-kube-api-access-9pcrs\") pod \"payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.725187 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.725187 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d071f4-6d6e-4052-912e-901053f6d5bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.725862 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.725843 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d071f4-6d6e-4052-912e-901053f6d5bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.733772 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.733743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcrs\" (UniqueName: \"kubernetes.io/projected/c7d071f4-6d6e-4052-912e-901053f6d5bf-kube-api-access-9pcrs\") pod \"payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:02.879451 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:02.879364 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:03.030215 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:03.030181 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:47:03.030648 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:03.030454 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:47:03.031159 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:03.031141 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:47:03.158943 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:03.158914 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh"] Sep 10 15:47:03.160571 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:47:03.160546 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d071f4_6d6e_4052_912e_901053f6d5bf.slice/crio-6565ade27f14e464f0e572a9fb1283391dfbfb48eb189e8e55670a9b7440ff82 WatchSource:0}: Error finding container 6565ade27f14e464f0e572a9fb1283391dfbfb48eb189e8e55670a9b7440ff82: Status 404 returned error can't find the container with id 6565ade27f14e464f0e572a9fb1283391dfbfb48eb189e8e55670a9b7440ff82 Sep 10 15:47:03.198925 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:03.198889 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" event={"ID":"c7d071f4-6d6e-4052-912e-901053f6d5bf","Type":"ContainerStarted","Data":"6565ade27f14e464f0e572a9fb1283391dfbfb48eb189e8e55670a9b7440ff82"} Sep 10 15:47:04.125128 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:04.125099 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:47:04.207137 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:04.207101 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" event={"ID":"c7d071f4-6d6e-4052-912e-901053f6d5bf","Type":"ContainerStarted","Data":"08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d"} Sep 10 15:47:04.234543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:04.234488 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" podStartSLOduration=2.234470653 podStartE2EDuration="2.234470653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:04.232672399 +0000 UTC m=+1194.015885464" watchObservedRunningTime="2026-09-10 15:47:04.234470653 +0000 UTC m=+1194.017683718" Sep 10 15:47:05.132199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:05.132169 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:47:05.213730 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:05.213701 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:16.222388 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:16.222352 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:47:36.228493 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.228460 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-ac72f9"] Sep 10 15:47:36.232239 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.232222 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:36.241306 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.241278 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wtngs\"" Sep 10 15:47:36.244271 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.244243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-ac72f9"] Sep 10 15:47:36.256796 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.256770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ccf\" (UniqueName: \"kubernetes.io/projected/e71b8bd2-7bce-490b-86a3-2f8d69daccca-kube-api-access-w4ccf\") pod \"test-curl-7749-ac72f9\" (UID: \"e71b8bd2-7bce-490b-86a3-2f8d69daccca\") " pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:36.357496 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.357466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ccf\" (UniqueName: \"kubernetes.io/projected/e71b8bd2-7bce-490b-86a3-2f8d69daccca-kube-api-access-w4ccf\") pod \"test-curl-7749-ac72f9\" (UID: \"e71b8bd2-7bce-490b-86a3-2f8d69daccca\") " pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:36.372024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.371976 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ccf\" (UniqueName: \"kubernetes.io/projected/e71b8bd2-7bce-490b-86a3-2f8d69daccca-kube-api-access-w4ccf\") pod \"test-curl-7749-ac72f9\" (UID: \"e71b8bd2-7bce-490b-86a3-2f8d69daccca\") " pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:36.548262 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.548232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:36.816863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.816834 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-ac72f9"] Sep 10 15:47:36.819506 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:47:36.819479 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode71b8bd2_7bce_490b_86a3_2f8d69daccca.slice/crio-827c292d523145e32330d47b61029567d25fb674ab4bc61a5810fb861ddec97f WatchSource:0}: Error finding container 827c292d523145e32330d47b61029567d25fb674ab4bc61a5810fb861ddec97f: Status 404 returned error can't find the container with id 827c292d523145e32330d47b61029567d25fb674ab4bc61a5810fb861ddec97f Sep 10 15:47:36.821729 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:36.821715 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:47:37.403288 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:37.403248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-ac72f9" event={"ID":"e71b8bd2-7bce-490b-86a3-2f8d69daccca","Type":"ContainerStarted","Data":"827c292d523145e32330d47b61029567d25fb674ab4bc61a5810fb861ddec97f"} Sep 10 15:47:38.411436 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:38.411401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-ac72f9" event={"ID":"e71b8bd2-7bce-490b-86a3-2f8d69daccca","Type":"ContainerStarted","Data":"008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f"} Sep 10 15:47:38.427953 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:38.427903 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7749-ac72f9" podStartSLOduration=1.690032858 podStartE2EDuration="2.427890222s" totalImagesPullingTime="737.857364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:38.425581627 +0000 UTC m=+1228.208794692" watchObservedRunningTime="2026-09-10 15:47:38.427890222 +0000 UTC m=+1228.211103285" Sep 10 15:47:39.022838 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.022802 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-ac72f9"] Sep 10 15:47:39.028592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.028561 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-ac72f9"] Sep 10 15:47:39.417857 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:47:39.417823 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7749-ac72f9" container="test-curl-7749-ac72f9" Sep 10 15:47:39.914061 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.914020 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx"] Sep 10 15:47:39.914680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.914663 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" containerName="test-curl-7749-ac72f9" Sep 10 15:47:39.914733 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.914683 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" containerName="test-curl-7749-ac72f9" Sep 10 15:47:39.914804 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.914793 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" containerName="test-curl-7749-ac72f9" Sep 10 15:47:39.917362 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.917335 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.919735 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.919711 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81f538c6-openshift-default-dockercfg-dcgbj\"" Sep 10 15:47:39.930032 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.930006 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx"] Sep 10 15:47:39.992226 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-socket\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-envoy\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-data\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-token\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992419 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-credential-socket\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992576 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-podinfo\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992576 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992515 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nhx\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-kube-api-access-84nhx\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992576 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d923c-5d6d-4d40-acb4-928113cf6386-istiod-ca-cert\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:39.992675 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:39.992627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-certs\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.093909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.093872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-certs\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.093979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-socket\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-envoy\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-data\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094110 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-token\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-credential-socket\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094192 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-podinfo\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nhx\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-kube-api-access-84nhx\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d923c-5d6d-4d40-acb4-928113cf6386-istiod-ca-cert\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.094856 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.094830 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-data\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.095283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.095180 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d923c-5d6d-4d40-acb4-928113cf6386-istiod-ca-cert\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.095423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.095189 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-socket\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.095544 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.095202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-credential-socket\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.095544 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.095411 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-certs\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.097890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.097866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-podinfo\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.097890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.097877 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-envoy\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.104449 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.104400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-token\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.104979 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.104954 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nhx\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-kube-api-access-84nhx\") pod \"e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.229060 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.228939 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:40.429530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.425595 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7749-ac72f9" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" containerName="test-curl-7749-ac72f9" containerID="cri-o://008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f" gracePeriod=2 Sep 10 15:47:40.429530 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:47:40.429227 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-ac72f9\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:40.512319 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.512288 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx"] Sep 10 15:47:40.514719 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:47:40.514686 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9d923c_5d6d_4d40_acb4_928113cf6386.slice/crio-85ce76489d0ee06f5118411d9a0313cde7f753f1abd89a22a3e68561508885a0 WatchSource:0}: Error finding container 85ce76489d0ee06f5118411d9a0313cde7f753f1abd89a22a3e68561508885a0: Status 404 returned error can't find the container with id 85ce76489d0ee06f5118411d9a0313cde7f753f1abd89a22a3e68561508885a0 Sep 10 15:47:40.517183 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.517145 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:47:40.517284 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.517216 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:47:40.517284 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:40.517247 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:47:40.786550 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:47:40.786453 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-ac72f9\" is forbidden: User \"system:node:ip-10-0-141-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.ec2.internal' and this object" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:41.432461 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:41.432422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" event={"ID":"6f9d923c-5d6d-4d40-acb4-928113cf6386","Type":"ContainerStarted","Data":"79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc"} Sep 10 15:47:41.432952 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:41.432479 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" event={"ID":"6f9d923c-5d6d-4d40-acb4-928113cf6386","Type":"ContainerStarted","Data":"85ce76489d0ee06f5118411d9a0313cde7f753f1abd89a22a3e68561508885a0"} Sep 10 15:47:41.452859 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:41.452812 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podStartSLOduration=2.452796794 podStartE2EDuration="2.452796794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:41.450877184 +0000 UTC m=+1231.234090247" watchObservedRunningTime="2026-09-10 15:47:41.452796794 +0000 UTC m=+1231.236009858" Sep 10 15:47:42.229879 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.229839 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:42.235401 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.235371 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:42.438762 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.438303 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:42.439487 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.439464 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:47:42.618721 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.618695 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:42.722455 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.722423 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4ccf\" (UniqueName: \"kubernetes.io/projected/e71b8bd2-7bce-490b-86a3-2f8d69daccca-kube-api-access-w4ccf\") pod \"e71b8bd2-7bce-490b-86a3-2f8d69daccca\" (UID: \"e71b8bd2-7bce-490b-86a3-2f8d69daccca\") " Sep 10 15:47:42.724601 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.724570 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b8bd2-7bce-490b-86a3-2f8d69daccca-kube-api-access-w4ccf" (OuterVolumeSpecName: "kube-api-access-w4ccf") pod "e71b8bd2-7bce-490b-86a3-2f8d69daccca" (UID: "e71b8bd2-7bce-490b-86a3-2f8d69daccca"). InnerVolumeSpecName "kube-api-access-w4ccf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:42.789012 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.788923 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" path="/var/lib/kubelet/pods/e71b8bd2-7bce-490b-86a3-2f8d69daccca/volumes" Sep 10 15:47:42.823406 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:42.823375 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ccf\" (UniqueName: \"kubernetes.io/projected/e71b8bd2-7bce-490b-86a3-2f8d69daccca-kube-api-access-w4ccf\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:47:43.444847 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:43.444814 2581 generic.go:356] "Generic (PLEG): container finished" podID="e71b8bd2-7bce-490b-86a3-2f8d69daccca" containerID="008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f" exitCode=137 Sep 10 15:47:43.445276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:43.444859 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-ac72f9" Sep 10 15:47:43.445276 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:43.444915 2581 scope.go:117] "RemoveContainer" containerID="008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f" Sep 10 15:47:43.454413 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:43.454392 2581 scope.go:117] "RemoveContainer" containerID="008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f" Sep 10 15:47:43.454669 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:47:43.454650 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f\": container with ID starting with 008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f not found: ID does not exist" containerID="008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f" Sep 10 15:47:43.454756 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:43.454676 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f"} err="failed to get container status \"008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f\": rpc error: code = NotFound desc = could not find container \"008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f\": container with ID starting with 008d66d4de9186d54189b78dfcaba2213f78b25d3fac42cf6767fc64f2c1363f not found: ID does not exist" Sep 10 15:47:47.938942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:47.938903 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8"] Sep 10 15:47:47.942480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:47.942459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:47.944865 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:47.944842 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81f538c6\"" Sep 10 15:47:47.954414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:47.954391 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8"] Sep 10 15:47:48.077310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.077266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pp5\" (UniqueName: \"kubernetes.io/projected/f566b019-f3d5-4e91-874b-9ef76e98f416-kube-api-access-m9pp5\") pod \"payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.077495 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.077337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f566b019-f3d5-4e91-874b-9ef76e98f416-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.178514 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.178478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pp5\" (UniqueName: \"kubernetes.io/projected/f566b019-f3d5-4e91-874b-9ef76e98f416-kube-api-access-m9pp5\") pod \"payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.178514 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.178528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f566b019-f3d5-4e91-874b-9ef76e98f416-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.179267 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.179246 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f566b019-f3d5-4e91-874b-9ef76e98f416-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.188480 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.188456 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pp5\" (UniqueName: \"kubernetes.io/projected/f566b019-f3d5-4e91-874b-9ef76e98f416-kube-api-access-m9pp5\") pod \"payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.253943 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.253857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:48.531318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:48.531292 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8"] Sep 10 15:47:48.532281 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:47:48.532253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf566b019_f3d5_4e91_874b_9ef76e98f416.slice/crio-13384c2171ccc129d682234e94465da499a23e18f4b33bb2faf4c9adc3148cad WatchSource:0}: Error finding container 13384c2171ccc129d682234e94465da499a23e18f4b33bb2faf4c9adc3148cad: Status 404 returned error can't find the container with id 13384c2171ccc129d682234e94465da499a23e18f4b33bb2faf4c9adc3148cad Sep 10 15:47:49.484965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:49.484926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" event={"ID":"f566b019-f3d5-4e91-874b-9ef76e98f416","Type":"ContainerStarted","Data":"0f3058deeabce6ed05aa9bfcd2c5609e0b7370719aefe32434fcc59f81fb867c"} Sep 10 15:47:49.485417 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:49.485028 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" event={"ID":"f566b019-f3d5-4e91-874b-9ef76e98f416","Type":"ContainerStarted","Data":"13384c2171ccc129d682234e94465da499a23e18f4b33bb2faf4c9adc3148cad"} Sep 10 15:47:49.485417 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:49.485140 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:47:49.505371 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:47:49.505323 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" podStartSLOduration=2.505304967 podStartE2EDuration="2.505304967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:49.502666399 +0000 UTC m=+1239.285879465" watchObservedRunningTime="2026-09-10 15:47:49.505304967 +0000 UTC m=+1239.288518032" Sep 10 15:48:00.493956 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:00.493873 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:48:03.574915 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:03.574830 2581 generic.go:356] "Generic (PLEG): container finished" podID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerID="79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc" exitCode=0 Sep 10 15:48:03.575429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:03.574910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" event={"ID":"6f9d923c-5d6d-4d40-acb4-928113cf6386","Type":"ContainerDied","Data":"79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc"} Sep 10 15:48:03.575501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:03.575449 2581 scope.go:117] "RemoveContainer" containerID="79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc" Sep 10 15:48:03.576650 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:03.576620 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:48:03.576737 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:03.576696 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:48:03.576737 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:03.576729 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:48:04.583104 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:04.583064 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" event={"ID":"6f9d923c-5d6d-4d40-acb4-928113cf6386","Type":"ContainerStarted","Data":"199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181"} Sep 10 15:48:05.229709 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:05.229674 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:48:05.229891 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:05.229866 2581 patch_prober.go:28] interesting pod/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 15:48:05.229937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:05.229907 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 15:48:06.229241 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:06.229204 2581 patch_prober.go:28] interesting pod/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 15:48:06.229649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:06.229257 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 15:48:07.229680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:07.229647 2581 patch_prober.go:28] interesting pod/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 15:48:07.230153 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:07.229706 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 15:48:08.229817 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:08.229778 2581 patch_prober.go:28] interesting pod/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 15:48:08.230334 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:08.229840 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 15:48:09.229898 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:09.229860 2581 patch_prober.go:28] interesting pod/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 15:48:09.230304 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:09.229915 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 15:48:10.234013 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:10.233969 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:48:10.234385 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:10.234322 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:48:10.234922 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:10.234904 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:48:10.813913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:10.813878 2581 scope.go:117] "RemoveContainer" containerID="23280b6a706396b1791dbb2c992613d157515905f571b068781a69ff64671f89" Sep 10 15:48:57.903051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:57.903018 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85"] Sep 10 15:48:57.905874 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:57.905857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:48:57.909397 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:57.909371 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:48:57.909397 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:57.909375 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:48:57.914468 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:57.914448 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85"] Sep 10 15:48:57.989572 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:57.989539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81f538c6-r2t85\" (UID: \"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:48:58.090822 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.090784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81f538c6-r2t85\" (UID: \"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:48:58.091452 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.091428 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81f538c6-r2t85\" (UID: \"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:48:58.217232 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.217149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:48:58.488791 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.488761 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85"] Sep 10 15:48:58.490408 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:48:58.490373 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a919bd6_d9bc_4deb_8dd7_07dc7ba2f774.slice/crio-c5953171a727820ffdbd670ba4ed9b27a9153182b4fa6aebd78b346f85239251 WatchSource:0}: Error finding container c5953171a727820ffdbd670ba4ed9b27a9153182b4fa6aebd78b346f85239251: Status 404 returned error can't find the container with id c5953171a727820ffdbd670ba4ed9b27a9153182b4fa6aebd78b346f85239251 Sep 10 15:48:58.903377 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.903346 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" event={"ID":"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774","Type":"ContainerStarted","Data":"a47b6ee7bc22c69cb0dedf53c30de2af71009709a1012a82c5c3781877827a51"} Sep 10 15:48:58.903768 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.903404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" event={"ID":"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774","Type":"ContainerStarted","Data":"c5953171a727820ffdbd670ba4ed9b27a9153182b4fa6aebd78b346f85239251"} Sep 10 15:48:58.924194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:58.924144 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" podStartSLOduration=1.9241291600000001 podStartE2EDuration="1.92412916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:58.922799324 +0000 UTC m=+1308.706012388" watchObservedRunningTime="2026-09-10 15:48:58.92412916 +0000 UTC m=+1308.707342258" Sep 10 15:48:59.910298 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:59.910260 2581 generic.go:356] "Generic (PLEG): container finished" podID="2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774" containerID="a47b6ee7bc22c69cb0dedf53c30de2af71009709a1012a82c5c3781877827a51" exitCode=0 Sep 10 15:48:59.910669 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:48:59.910336 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" event={"ID":"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774","Type":"ContainerDied","Data":"a47b6ee7bc22c69cb0dedf53c30de2af71009709a1012a82c5c3781877827a51"} Sep 10 15:49:01.053161 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.053141 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:49:01.121551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.121520 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774-maas-api-service-ca\") pod \"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774\" (UID: \"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774\") " Sep 10 15:49:01.121861 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.121839 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774" (UID: "2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:49:01.223230 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.223149 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:49:01.922353 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.922319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" event={"ID":"2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774","Type":"ContainerDied","Data":"c5953171a727820ffdbd670ba4ed9b27a9153182b4fa6aebd78b346f85239251"} Sep 10 15:49:01.922526 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.922365 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5953171a727820ffdbd670ba4ed9b27a9153182b4fa6aebd78b346f85239251" Sep 10 15:49:01.922526 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:01.922335 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85" Sep 10 15:49:10.840524 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:10.840491 2581 scope.go:117] "RemoveContainer" containerID="0f5f9f778fae23f247262ad557d8f9a47badff10ce8578b052372e3ed5b74f82" Sep 10 15:49:13.815606 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:13.815561 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8"] Sep 10 15:49:13.816085 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:13.815808 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" podUID="f566b019-f3d5-4e91-874b-9ef76e98f416" containerName="payload-pre-processing" containerID="cri-o://0f3058deeabce6ed05aa9bfcd2c5609e0b7370719aefe32434fcc59f81fb867c" gracePeriod=30 Sep 10 15:49:13.994900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:13.994864 2581 generic.go:356] "Generic (PLEG): container finished" podID="f566b019-f3d5-4e91-874b-9ef76e98f416" containerID="0f3058deeabce6ed05aa9bfcd2c5609e0b7370719aefe32434fcc59f81fb867c" exitCode=0 Sep 10 15:49:13.995046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:13.994940 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" event={"ID":"f566b019-f3d5-4e91-874b-9ef76e98f416","Type":"ContainerDied","Data":"0f3058deeabce6ed05aa9bfcd2c5609e0b7370719aefe32434fcc59f81fb867c"} Sep 10 15:49:14.012087 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.012053 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85"] Sep 10 15:49:14.015672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.015640 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f538c6-r2t85"] Sep 10 15:49:14.067705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.067646 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:49:14.161481 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.161438 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f566b019-f3d5-4e91-874b-9ef76e98f416-plugins-config-volume\") pod \"f566b019-f3d5-4e91-874b-9ef76e98f416\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " Sep 10 15:49:14.161669 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.161505 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9pp5\" (UniqueName: \"kubernetes.io/projected/f566b019-f3d5-4e91-874b-9ef76e98f416-kube-api-access-m9pp5\") pod \"f566b019-f3d5-4e91-874b-9ef76e98f416\" (UID: \"f566b019-f3d5-4e91-874b-9ef76e98f416\") " Sep 10 15:49:14.161975 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.161847 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f566b019-f3d5-4e91-874b-9ef76e98f416-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f566b019-f3d5-4e91-874b-9ef76e98f416" (UID: "f566b019-f3d5-4e91-874b-9ef76e98f416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:49:14.167207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.167174 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f566b019-f3d5-4e91-874b-9ef76e98f416-kube-api-access-m9pp5" (OuterVolumeSpecName: "kube-api-access-m9pp5") pod "f566b019-f3d5-4e91-874b-9ef76e98f416" (UID: "f566b019-f3d5-4e91-874b-9ef76e98f416"). InnerVolumeSpecName "kube-api-access-m9pp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:49:14.262441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.262394 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f566b019-f3d5-4e91-874b-9ef76e98f416-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:49:14.262441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.262433 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9pp5\" (UniqueName: \"kubernetes.io/projected/f566b019-f3d5-4e91-874b-9ef76e98f416-kube-api-access-m9pp5\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:49:14.788382 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:14.788347 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774" path="/var/lib/kubelet/pods/2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774/volumes" Sep 10 15:49:15.001754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:15.001723 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" Sep 10 15:49:15.001754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:15.001731 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8" event={"ID":"f566b019-f3d5-4e91-874b-9ef76e98f416","Type":"ContainerDied","Data":"13384c2171ccc129d682234e94465da499a23e18f4b33bb2faf4c9adc3148cad"} Sep 10 15:49:15.002317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:15.001785 2581 scope.go:117] "RemoveContainer" containerID="0f3058deeabce6ed05aa9bfcd2c5609e0b7370719aefe32434fcc59f81fb867c" Sep 10 15:49:15.020826 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:15.020800 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8"] Sep 10 15:49:15.026546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:15.026523 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f538c6-897d66486-8vgn8"] Sep 10 15:49:16.793173 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:16.790793 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f566b019-f3d5-4e91-874b-9ef76e98f416" path="/var/lib/kubelet/pods/f566b019-f3d5-4e91-874b-9ef76e98f416/volumes" Sep 10 15:49:46.609215 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609135 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7955ffc5d9-rxrpm"] Sep 10 15:49:46.609676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609587 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f566b019-f3d5-4e91-874b-9ef76e98f416" containerName="payload-pre-processing" Sep 10 15:49:46.609676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609599 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f566b019-f3d5-4e91-874b-9ef76e98f416" containerName="payload-pre-processing" Sep 10 15:49:46.609676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609613 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774" containerName="revoke-keys" Sep 10 15:49:46.609676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609619 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774" containerName="revoke-keys" Sep 10 15:49:46.609808 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609682 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a919bd6-d9bc-4deb-8dd7-07dc7ba2f774" containerName="revoke-keys" Sep 10 15:49:46.609808 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.609696 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f566b019-f3d5-4e91-874b-9ef76e98f416" containerName="payload-pre-processing" Sep 10 15:49:46.611930 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.611911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.614368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.614339 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 15:49:46.614523 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.614506 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 15:49:46.615248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.615211 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 15:49:46.615361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.615226 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 15:49:46.615361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.615268 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 15:49:46.615361 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.615340 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-57xrq\"" Sep 10 15:49:46.619083 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.619061 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 15:49:46.623856 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.623834 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7955ffc5d9-rxrpm"] Sep 10 15:49:46.685302 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-oauth-config\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.685473 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9kw\" (UniqueName: \"kubernetes.io/projected/191083eb-1f95-4ab1-ae6a-5facd2397e5d-kube-api-access-ct9kw\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.685473 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-trusted-ca-bundle\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.685473 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-service-ca\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.685473 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-serving-cert\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.685612 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685495 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-config\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.685612 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.685509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-oauth-serving-cert\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.786743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9kw\" (UniqueName: \"kubernetes.io/projected/191083eb-1f95-4ab1-ae6a-5facd2397e5d-kube-api-access-ct9kw\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.786743 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786751 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-trusted-ca-bundle\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-service-ca\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-serving-cert\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-config\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-oauth-serving-cert\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787026 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.786923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-oauth-config\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787685 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.787656 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-config\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787853 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.787736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-oauth-serving-cert\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.787947 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.787769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-trusted-ca-bundle\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.788227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.788206 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/191083eb-1f95-4ab1-ae6a-5facd2397e5d-service-ca\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.789629 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.789610 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-serving-cert\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.789723 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.789628 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/191083eb-1f95-4ab1-ae6a-5facd2397e5d-console-oauth-config\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.796886 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.796866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9kw\" (UniqueName: \"kubernetes.io/projected/191083eb-1f95-4ab1-ae6a-5facd2397e5d-kube-api-access-ct9kw\") pod \"console-7955ffc5d9-rxrpm\" (UID: \"191083eb-1f95-4ab1-ae6a-5facd2397e5d\") " pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:46.922547 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:46.922463 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:47.199423 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:47.199343 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7955ffc5d9-rxrpm"] Sep 10 15:49:47.202638 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:49:47.202613 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191083eb_1f95_4ab1_ae6a_5facd2397e5d.slice/crio-383e4e528ea802c14b38fe7410c0692dcf29e8b8f5b74c0fb3aca2f3981dabba WatchSource:0}: Error finding container 383e4e528ea802c14b38fe7410c0692dcf29e8b8f5b74c0fb3aca2f3981dabba: Status 404 returned error can't find the container with id 383e4e528ea802c14b38fe7410c0692dcf29e8b8f5b74c0fb3aca2f3981dabba Sep 10 15:49:48.198442 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:48.198404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7955ffc5d9-rxrpm" event={"ID":"191083eb-1f95-4ab1-ae6a-5facd2397e5d","Type":"ContainerStarted","Data":"8270eaaba5dd926c5a35b8d5d94c3de7db86c3b55d0d3a7b5cd09f853c492542"} Sep 10 15:49:48.198830 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:48.198458 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7955ffc5d9-rxrpm" event={"ID":"191083eb-1f95-4ab1-ae6a-5facd2397e5d","Type":"ContainerStarted","Data":"383e4e528ea802c14b38fe7410c0692dcf29e8b8f5b74c0fb3aca2f3981dabba"} Sep 10 15:49:48.222334 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:48.222287 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7955ffc5d9-rxrpm" podStartSLOduration=2.222270815 podStartE2EDuration="2.222270815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:48.220020434 +0000 UTC m=+1358.003233500" watchObservedRunningTime="2026-09-10 15:49:48.222270815 +0000 UTC m=+1358.005483880" Sep 10 15:49:56.923268 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:56.923233 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:56.923646 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:56.923290 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:56.928384 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:56.928360 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:49:57.254927 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:49:57.254847 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7955ffc5d9-rxrpm" Sep 10 15:50:10.862750 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:10.862718 2581 scope.go:117] "RemoveContainer" containerID="40b055869c3d4e5e8c5ef8a50453a4913dbc05382294619a7760a140cc14e82e" Sep 10 15:50:10.871847 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:10.871825 2581 scope.go:117] "RemoveContainer" containerID="c54c559f1f324c50e4827cf5d5b99f53f12ed748ad4b9978d16fa96d881074d6" Sep 10 15:50:10.881360 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:10.881341 2581 scope.go:117] "RemoveContainer" containerID="728206c6c03adc14f4f2da8542ef4315ad54f0184cc713f36f748130148e6b67" Sep 10 15:50:12.997628 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:12.997587 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx"] Sep 10 15:50:12.998074 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:12.997828 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" containerID="cri-o://199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181" gracePeriod=30 Sep 10 15:50:18.259553 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.259492 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:50:18.389797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.389759 2581 generic.go:356] "Generic (PLEG): container finished" podID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerID="199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181" exitCode=0 Sep 10 15:50:18.389965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.389834 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" Sep 10 15:50:18.389965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.389843 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" event={"ID":"6f9d923c-5d6d-4d40-acb4-928113cf6386","Type":"ContainerDied","Data":"199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181"} Sep 10 15:50:18.389965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.389941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx" event={"ID":"6f9d923c-5d6d-4d40-acb4-928113cf6386","Type":"ContainerDied","Data":"85ce76489d0ee06f5118411d9a0313cde7f753f1abd89a22a3e68561508885a0"} Sep 10 15:50:18.390103 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.389972 2581 scope.go:117] "RemoveContainer" containerID="199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181" Sep 10 15:50:18.403626 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.403606 2581 scope.go:117] "RemoveContainer" containerID="79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc" Sep 10 15:50:18.411789 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.411760 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-podinfo\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.411917 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.411803 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-socket\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.411917 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.411852 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-credential-socket\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.411915 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-certs\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.411962 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-envoy\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412037 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d923c-5d6d-4d40-acb4-928113cf6386-istiod-ca-cert\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412080 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-token\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412123 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-data\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412155 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84nhx\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-kube-api-access-84nhx\") pod \"6f9d923c-5d6d-4d40-acb4-928113cf6386\" (UID: \"6f9d923c-5d6d-4d40-acb4-928113cf6386\") " Sep 10 15:50:18.412213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412177 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.412213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412187 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.412492 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412215 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.412492 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412368 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.412661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412624 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.412661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412650 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.412806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412665 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.412806 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412677 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.412913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.412821 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9d923c-5d6d-4d40-acb4-928113cf6386-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:50:18.413507 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.413482 2581 scope.go:117] "RemoveContainer" containerID="199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181" Sep 10 15:50:18.413878 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:50:18.413856 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181\": container with ID starting with 199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181 not found: ID does not exist" containerID="199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181" Sep 10 15:50:18.413967 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.413890 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181"} err="failed to get container status \"199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181\": rpc error: code = NotFound desc = could not find container \"199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181\": container with ID starting with 199f586ca540a8ba6ce6a58dc7e2add3f88d5b6a7a35f40ddcddea925cbdf181 not found: ID does not exist" Sep 10 15:50:18.413967 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.413916 2581 scope.go:117] "RemoveContainer" containerID="79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc" Sep 10 15:50:18.414263 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.414231 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:50:18.414353 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:50:18.414330 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc\": container with ID starting with 79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc not found: ID does not exist" containerID="79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc" Sep 10 15:50:18.414398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.414363 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:18.414398 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.414362 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc"} err="failed to get container status \"79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc\": rpc error: code = NotFound desc = could not find container \"79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc\": container with ID starting with 79a96c0748b2a7474b12bb70c4bdc8731830a9f0c221456d1ba9685e98169fbc not found: ID does not exist" Sep 10 15:50:18.414706 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.414680 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.414793 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.414744 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-kube-api-access-84nhx" (OuterVolumeSpecName: "kube-api-access-84nhx") pod "6f9d923c-5d6d-4d40-acb4-928113cf6386" (UID: "6f9d923c-5d6d-4d40-acb4-928113cf6386"). InnerVolumeSpecName "kube-api-access-84nhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:18.513396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.513302 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.513396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.513334 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d923c-5d6d-4d40-acb4-928113cf6386-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.513396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.513346 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.513396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.513357 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nhx\" (UniqueName: \"kubernetes.io/projected/6f9d923c-5d6d-4d40-acb4-928113cf6386-kube-api-access-84nhx\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.513396 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.513370 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d923c-5d6d-4d40-acb4-928113cf6386-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.713910 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.713878 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx"] Sep 10 15:50:18.717310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.717285 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81f538c6-openshift-default-66bc96cfcb-dgrsx"] Sep 10 15:50:18.788786 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:50:18.788706 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" path="/var/lib/kubelet/pods/6f9d923c-5d6d-4d40-acb4-928113cf6386/volumes" Sep 10 15:51:09.875188 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.875156 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr"] Sep 10 15:51:09.875591 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.875584 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" Sep 10 15:51:09.875633 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.875595 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" Sep 10 15:51:09.875683 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.875672 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" Sep 10 15:51:09.875723 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.875683 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" Sep 10 15:51:09.877839 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.877824 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.880426 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.880402 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1c1464-openshift-default-dockercfg-szhnd\"" Sep 10 15:51:09.896882 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.896857 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr"] Sep 10 15:51:09.914430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75qt\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-kube-api-access-s75qt\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-podinfo\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-token\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914588 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-envoy\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914760 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-certs\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914823 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-socket\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914823 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-credential-socket\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914929 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914903 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istiod-ca-cert\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:09.914971 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:09.914951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-data\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.015958 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.015917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istiod-ca-cert\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.015958 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.015957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-data\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.015985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75qt\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-kube-api-access-s75qt\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-podinfo\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-token\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-envoy\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016165 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-certs\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016216 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016213 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-socket\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016500 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-credential-socket\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.016924 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016885 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-data\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.017073 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.016956 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-socket\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.017198 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.017172 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istiod-ca-cert\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.017357 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.017194 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-certs\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.019049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.019017 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-envoy\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.022294 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.022266 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-credential-socket\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.026512 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.026489 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-token\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.026607 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.026572 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-podinfo\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.027971 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.027950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75qt\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-kube-api-access-s75qt\") pod \"e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.188757 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.188660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:10.467352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.467325 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr"] Sep 10 15:51:10.468885 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:51:10.468856 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f9e174_6fce_49d7_81c1_690ce6dfcfdc.slice/crio-0e1c0c7ec2aaddc3d7d14229f81caa6562e2e36ceee5f1fdeee524bfda546724 WatchSource:0}: Error finding container 0e1c0c7ec2aaddc3d7d14229f81caa6562e2e36ceee5f1fdeee524bfda546724: Status 404 returned error can't find the container with id 0e1c0c7ec2aaddc3d7d14229f81caa6562e2e36ceee5f1fdeee524bfda546724 Sep 10 15:51:10.471121 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.471085 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:51:10.471233 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.471148 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:51:10.471233 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.471178 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:51:10.692069 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.692030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" event={"ID":"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc","Type":"ContainerStarted","Data":"bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471"} Sep 10 15:51:10.692269 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.692092 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" event={"ID":"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc","Type":"ContainerStarted","Data":"0e1c0c7ec2aaddc3d7d14229f81caa6562e2e36ceee5f1fdeee524bfda546724"} Sep 10 15:51:10.712161 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.712086 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podStartSLOduration=1.712069691 podStartE2EDuration="1.712069691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:10.711199266 +0000 UTC m=+1440.494412330" watchObservedRunningTime="2026-09-10 15:51:10.712069691 +0000 UTC m=+1440.495282755" Sep 10 15:51:10.917417 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.917376 2581 scope.go:117] "RemoveContainer" containerID="0add825e498da10ee7726d854fc191472d0cb26f576abb7e8e99e59b5873db52" Sep 10 15:51:10.926056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.926033 2581 scope.go:117] "RemoveContainer" containerID="63f64763f93b7196b0ab98c8674fbdaf9d5c9b0449daacb7cd2f9ca54a66feed" Sep 10 15:51:10.933985 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:10.933963 2581 scope.go:117] "RemoveContainer" containerID="4a3257b12cb8b4719f3cd7b530ab22574b29e996dce58c84dc004670317381bf" Sep 10 15:51:11.188917 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:11.188830 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:11.195025 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:11.194982 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:11.697775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:11.697740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:11.698822 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:11.698802 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:17.114096 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.114058 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7"] Sep 10 15:51:17.114569 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.114552 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" Sep 10 15:51:17.114629 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.114571 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9d923c-5d6d-4d40-acb4-928113cf6386" containerName="istio-proxy" Sep 10 15:51:17.117871 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.117849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.125931 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.125914 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1c1464-dockercfg-z5ht2\"" Sep 10 15:51:17.126181 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.126166 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1c1464\"" Sep 10 15:51:17.131264 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.131243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7"] Sep 10 15:51:17.185132 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.185102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aae957f-e498-4bd3-843d-f4423a6c9d43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.185295 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.185164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfxl\" (UniqueName: \"kubernetes.io/projected/6aae957f-e498-4bd3-843d-f4423a6c9d43-kube-api-access-bvfxl\") pod \"payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.286358 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.286320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aae957f-e498-4bd3-843d-f4423a6c9d43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.286524 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.286395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfxl\" (UniqueName: \"kubernetes.io/projected/6aae957f-e498-4bd3-843d-f4423a6c9d43-kube-api-access-bvfxl\") pod \"payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.286982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.286961 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aae957f-e498-4bd3-843d-f4423a6c9d43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.296039 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.296010 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfxl\" (UniqueName: \"kubernetes.io/projected/6aae957f-e498-4bd3-843d-f4423a6c9d43-kube-api-access-bvfxl\") pod \"payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.429186 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.429098 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:17.701172 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.701148 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7"] Sep 10 15:51:17.703168 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:51:17.703141 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aae957f_e498_4bd3_843d_f4423a6c9d43.slice/crio-dd13cf37404ec098e8ebc94b76ee0fb226566591e64555cdf3cd2ea7ec9ee371 WatchSource:0}: Error finding container dd13cf37404ec098e8ebc94b76ee0fb226566591e64555cdf3cd2ea7ec9ee371: Status 404 returned error can't find the container with id dd13cf37404ec098e8ebc94b76ee0fb226566591e64555cdf3cd2ea7ec9ee371 Sep 10 15:51:17.733279 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:17.733253 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" event={"ID":"6aae957f-e498-4bd3-843d-f4423a6c9d43","Type":"ContainerStarted","Data":"dd13cf37404ec098e8ebc94b76ee0fb226566591e64555cdf3cd2ea7ec9ee371"} Sep 10 15:51:18.740079 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:18.740039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" event={"ID":"6aae957f-e498-4bd3-843d-f4423a6c9d43","Type":"ContainerStarted","Data":"13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed"} Sep 10 15:51:18.740518 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:18.740211 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:18.759564 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:18.759501 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" podStartSLOduration=1.7594792479999999 podStartE2EDuration="1.759479248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:18.756335054 +0000 UTC m=+1448.539548118" watchObservedRunningTime="2026-09-10 15:51:18.759479248 +0000 UTC m=+1448.542692313" Sep 10 15:51:29.748471 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:29.748440 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:51:33.838668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:33.838631 2581 generic.go:356] "Generic (PLEG): container finished" podID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerID="bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471" exitCode=0 Sep 10 15:51:33.839113 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:33.838706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" event={"ID":"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc","Type":"ContainerDied","Data":"bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471"} Sep 10 15:51:33.839246 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:33.839229 2581 scope.go:117] "RemoveContainer" containerID="bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471" Sep 10 15:51:33.840520 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:33.840489 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:51:33.840585 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:33.840563 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:51:33.840625 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:33.840593 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:51:34.846880 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:34.846844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" event={"ID":"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc","Type":"ContainerStarted","Data":"5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42"} Sep 10 15:51:35.189574 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:35.189485 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:35.189791 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:35.189771 2581 patch_prober.go:28] interesting pod/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 10 15:51:35.189849 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:35.189815 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 10 15:51:36.189840 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:36.189808 2581 patch_prober.go:28] interesting pod/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 10 15:51:36.190246 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:36.189860 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 10 15:51:37.188933 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:37.188900 2581 patch_prober.go:28] interesting pod/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 10 15:51:37.189124 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:37.188953 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 10 15:51:38.189888 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:38.189853 2581 patch_prober.go:28] interesting pod/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 10 15:51:38.190328 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:38.189924 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 10 15:51:39.189589 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:39.189545 2581 patch_prober.go:28] interesting pod/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 10 15:51:39.189768 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:39.189616 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 10 15:51:40.193759 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:40.193729 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:40.194256 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:40.194098 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:40.194668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:40.194646 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:51:51.889466 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:51.889428 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4"] Sep 10 15:51:51.892403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:51.892385 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:51.894823 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:51.894801 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-59c481\"" Sep 10 15:51:51.904427 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:51.904403 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4"] Sep 10 15:51:51.950967 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:51.950936 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc54\" (UniqueName: \"kubernetes.io/projected/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-kube-api-access-jgc54\") pod \"payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:51.951143 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:51.950975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.052359 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.052323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc54\" (UniqueName: \"kubernetes.io/projected/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-kube-api-access-jgc54\") pod \"payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.052529 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.052366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.053089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.053068 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.062482 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.062451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc54\" (UniqueName: \"kubernetes.io/projected/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-kube-api-access-jgc54\") pod \"payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.204194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.204110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.474678 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.474652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4"] Sep 10 15:51:52.476608 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:51:52.476580 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507f2db1_5470_4ddd_a9a6_ab5a92d47e16.slice/crio-133875825bfeffa041dc6e21645571d2718dca733e3be20b610710ed475ba5f8 WatchSource:0}: Error finding container 133875825bfeffa041dc6e21645571d2718dca733e3be20b610710ed475ba5f8: Status 404 returned error can't find the container with id 133875825bfeffa041dc6e21645571d2718dca733e3be20b610710ed475ba5f8 Sep 10 15:51:52.961056 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.961019 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" event={"ID":"507f2db1-5470-4ddd-a9a6-ab5a92d47e16","Type":"ContainerStarted","Data":"f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c"} Sep 10 15:51:52.961441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.961078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" event={"ID":"507f2db1-5470-4ddd-a9a6-ab5a92d47e16","Type":"ContainerStarted","Data":"133875825bfeffa041dc6e21645571d2718dca733e3be20b610710ed475ba5f8"} Sep 10 15:51:52.961441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.961102 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:51:52.982672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:51:52.982620 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" podStartSLOduration=1.982605928 podStartE2EDuration="1.982605928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:52.980043294 +0000 UTC m=+1482.763256358" watchObservedRunningTime="2026-09-10 15:51:52.982605928 +0000 UTC m=+1482.765818992" Sep 10 15:52:03.969894 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:03.969860 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:52:10.972672 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:10.972636 2581 scope.go:117] "RemoveContainer" containerID="39476d5a4a2eb914d58f32a9f4c62ad52b7f68b7aa82a930683122296370189a" Sep 10 15:52:19.595265 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.595232 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x"] Sep 10 15:52:19.598182 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.598165 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:19.600954 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.600931 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:52:19.601139 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.600938 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:52:19.607845 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.607823 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x"] Sep 10 15:52:19.640918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.640889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc078c4c-ef52-43c3-a959-6200909a14a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x\" (UID: \"bc078c4c-ef52-43c3-a959-6200909a14a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:19.742322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.742290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc078c4c-ef52-43c3-a959-6200909a14a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x\" (UID: \"bc078c4c-ef52-43c3-a959-6200909a14a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:19.742949 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.742921 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc078c4c-ef52-43c3-a959-6200909a14a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x\" (UID: \"bc078c4c-ef52-43c3-a959-6200909a14a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:19.910428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:19.910336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:20.182501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:20.182476 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x"] Sep 10 15:52:20.184807 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:52:20.184783 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc078c4c_ef52_43c3_a959_6200909a14a8.slice/crio-a7caf35c17903c82ded80b9ecd9ec596ec6a44b74fe23828730e673180f41eb8 WatchSource:0}: Error finding container a7caf35c17903c82ded80b9ecd9ec596ec6a44b74fe23828730e673180f41eb8: Status 404 returned error can't find the container with id a7caf35c17903c82ded80b9ecd9ec596ec6a44b74fe23828730e673180f41eb8 Sep 10 15:52:21.125962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:21.125930 2581 generic.go:356] "Generic (PLEG): container finished" podID="bc078c4c-ef52-43c3-a959-6200909a14a8" containerID="5c80140faf853e6c327950c22117211ddb2792d24cd60ca8c7290713392fe9c4" exitCode=0 Sep 10 15:52:21.126376 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:21.126030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" event={"ID":"bc078c4c-ef52-43c3-a959-6200909a14a8","Type":"ContainerDied","Data":"5c80140faf853e6c327950c22117211ddb2792d24cd60ca8c7290713392fe9c4"} Sep 10 15:52:21.126376 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:21.126079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" event={"ID":"bc078c4c-ef52-43c3-a959-6200909a14a8","Type":"ContainerStarted","Data":"a7caf35c17903c82ded80b9ecd9ec596ec6a44b74fe23828730e673180f41eb8"} Sep 10 15:52:22.255688 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:22.255659 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:22.367788 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:22.367750 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc078c4c-ef52-43c3-a959-6200909a14a8-maas-api-service-ca\") pod \"bc078c4c-ef52-43c3-a959-6200909a14a8\" (UID: \"bc078c4c-ef52-43c3-a959-6200909a14a8\") " Sep 10 15:52:22.368124 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:22.368102 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc078c4c-ef52-43c3-a959-6200909a14a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc078c4c-ef52-43c3-a959-6200909a14a8" (UID: "bc078c4c-ef52-43c3-a959-6200909a14a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:52:22.468910 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:22.468831 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc078c4c-ef52-43c3-a959-6200909a14a8-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:23.138661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:23.138576 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" event={"ID":"bc078c4c-ef52-43c3-a959-6200909a14a8","Type":"ContainerDied","Data":"a7caf35c17903c82ded80b9ecd9ec596ec6a44b74fe23828730e673180f41eb8"} Sep 10 15:52:23.138661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:23.138621 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x" Sep 10 15:52:23.138661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:23.138650 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7caf35c17903c82ded80b9ecd9ec596ec6a44b74fe23828730e673180f41eb8" Sep 10 15:52:35.219551 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.219452 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7"] Sep 10 15:52:35.220034 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.219691 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" podUID="6aae957f-e498-4bd3-843d-f4423a6c9d43" containerName="payload-processing" containerID="cri-o://13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed" gracePeriod=30 Sep 10 15:52:35.441249 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.440970 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x"] Sep 10 15:52:35.448424 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.448392 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1c1464-k5j4x"] Sep 10 15:52:35.486773 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.486751 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:52:35.599326 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.599286 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aae957f-e498-4bd3-843d-f4423a6c9d43-plugins-config-volume\") pod \"6aae957f-e498-4bd3-843d-f4423a6c9d43\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " Sep 10 15:52:35.599554 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.599413 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvfxl\" (UniqueName: \"kubernetes.io/projected/6aae957f-e498-4bd3-843d-f4423a6c9d43-kube-api-access-bvfxl\") pod \"6aae957f-e498-4bd3-843d-f4423a6c9d43\" (UID: \"6aae957f-e498-4bd3-843d-f4423a6c9d43\") " Sep 10 15:52:35.600066 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.600029 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aae957f-e498-4bd3-843d-f4423a6c9d43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6aae957f-e498-4bd3-843d-f4423a6c9d43" (UID: "6aae957f-e498-4bd3-843d-f4423a6c9d43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:52:35.601745 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.601717 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aae957f-e498-4bd3-843d-f4423a6c9d43-kube-api-access-bvfxl" (OuterVolumeSpecName: "kube-api-access-bvfxl") pod "6aae957f-e498-4bd3-843d-f4423a6c9d43" (UID: "6aae957f-e498-4bd3-843d-f4423a6c9d43"). InnerVolumeSpecName "kube-api-access-bvfxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:52:35.700703 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.700663 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvfxl\" (UniqueName: \"kubernetes.io/projected/6aae957f-e498-4bd3-843d-f4423a6c9d43-kube-api-access-bvfxl\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:35.700703 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:35.700695 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aae957f-e498-4bd3-843d-f4423a6c9d43-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:36.216654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.216620 2581 generic.go:356] "Generic (PLEG): container finished" podID="6aae957f-e498-4bd3-843d-f4423a6c9d43" containerID="13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed" exitCode=0 Sep 10 15:52:36.216654 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.216661 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" Sep 10 15:52:36.216890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.216674 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" event={"ID":"6aae957f-e498-4bd3-843d-f4423a6c9d43","Type":"ContainerDied","Data":"13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed"} Sep 10 15:52:36.216890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.216729 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7" event={"ID":"6aae957f-e498-4bd3-843d-f4423a6c9d43","Type":"ContainerDied","Data":"dd13cf37404ec098e8ebc94b76ee0fb226566591e64555cdf3cd2ea7ec9ee371"} Sep 10 15:52:36.216890 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.216759 2581 scope.go:117] "RemoveContainer" containerID="13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed" Sep 10 15:52:36.228640 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.227431 2581 scope.go:117] "RemoveContainer" containerID="13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed" Sep 10 15:52:36.229167 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:52:36.229143 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed\": container with ID starting with 13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed not found: ID does not exist" containerID="13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed" Sep 10 15:52:36.229473 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.229307 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed"} err="failed to get container status \"13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed\": rpc error: code = NotFound desc = could not find container \"13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed\": container with ID starting with 13787e15867eece6ba528accb4f0c08e475ae009eab1fdf821f8f1ab21ea6bed not found: ID does not exist" Sep 10 15:52:36.240634 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.240609 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7"] Sep 10 15:52:36.246089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.246064 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1c1464-5598658c45-xmjm7"] Sep 10 15:52:36.788799 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.788765 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aae957f-e498-4bd3-843d-f4423a6c9d43" path="/var/lib/kubelet/pods/6aae957f-e498-4bd3-843d-f4423a6c9d43/volumes" Sep 10 15:52:36.789151 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:36.789136 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc078c4c-ef52-43c3-a959-6200909a14a8" path="/var/lib/kubelet/pods/bc078c4c-ef52-43c3-a959-6200909a14a8/volumes" Sep 10 15:52:42.474807 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.474776 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr"] Sep 10 15:52:42.475280 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.475042 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" containerID="cri-o://5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42" gracePeriod=30 Sep 10 15:52:42.831774 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.831744 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz"] Sep 10 15:52:42.832208 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.832193 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc078c4c-ef52-43c3-a959-6200909a14a8" containerName="revoke-keys" Sep 10 15:52:42.832258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.832210 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc078c4c-ef52-43c3-a959-6200909a14a8" containerName="revoke-keys" Sep 10 15:52:42.832258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.832228 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aae957f-e498-4bd3-843d-f4423a6c9d43" containerName="payload-processing" Sep 10 15:52:42.832258 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.832233 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aae957f-e498-4bd3-843d-f4423a6c9d43" containerName="payload-processing" Sep 10 15:52:42.832368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.832294 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc078c4c-ef52-43c3-a959-6200909a14a8" containerName="revoke-keys" Sep 10 15:52:42.832368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.832310 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aae957f-e498-4bd3-843d-f4423a6c9d43" containerName="payload-processing" Sep 10 15:52:42.834649 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.834632 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:42.836778 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.836736 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:52:42.836964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.836949 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:52:42.845429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.845396 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz"] Sep 10 15:52:42.975776 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:42.975735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ad399-522b-4d12-a7f9-ba38df0c7131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59c481-dcgnz\" (UID: \"cd9ad399-522b-4d12-a7f9-ba38df0c7131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:43.077040 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:43.076984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ad399-522b-4d12-a7f9-ba38df0c7131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59c481-dcgnz\" (UID: \"cd9ad399-522b-4d12-a7f9-ba38df0c7131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:43.077638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:43.077605 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ad399-522b-4d12-a7f9-ba38df0c7131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59c481-dcgnz\" (UID: \"cd9ad399-522b-4d12-a7f9-ba38df0c7131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:43.157327 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:43.157247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:43.434580 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:43.434552 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz"] Sep 10 15:52:43.435987 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:52:43.435961 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9ad399_522b_4d12_a7f9_ba38df0c7131.slice/crio-d521d86390c5ceb374cb9641dafa1f9f07ab2c00e3eed29d6c7b37c0ec96f0ff WatchSource:0}: Error finding container d521d86390c5ceb374cb9641dafa1f9f07ab2c00e3eed29d6c7b37c0ec96f0ff: Status 404 returned error can't find the container with id d521d86390c5ceb374cb9641dafa1f9f07ab2c00e3eed29d6c7b37c0ec96f0ff Sep 10 15:52:44.262950 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:44.262908 2581 generic.go:356] "Generic (PLEG): container finished" podID="cd9ad399-522b-4d12-a7f9-ba38df0c7131" containerID="20f2c9286273c6dda7d235af48aa068be58453b6ded5e3e751c681123429ee0e" exitCode=0 Sep 10 15:52:44.263328 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:44.262976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" event={"ID":"cd9ad399-522b-4d12-a7f9-ba38df0c7131","Type":"ContainerDied","Data":"20f2c9286273c6dda7d235af48aa068be58453b6ded5e3e751c681123429ee0e"} Sep 10 15:52:44.263328 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:44.263047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" event={"ID":"cd9ad399-522b-4d12-a7f9-ba38df0c7131","Type":"ContainerStarted","Data":"d521d86390c5ceb374cb9641dafa1f9f07ab2c00e3eed29d6c7b37c0ec96f0ff"} Sep 10 15:52:45.408275 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:45.408248 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:45.504771 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:45.504737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ad399-522b-4d12-a7f9-ba38df0c7131-maas-api-service-ca\") pod \"cd9ad399-522b-4d12-a7f9-ba38df0c7131\" (UID: \"cd9ad399-522b-4d12-a7f9-ba38df0c7131\") " Sep 10 15:52:45.505170 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:45.505145 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9ad399-522b-4d12-a7f9-ba38df0c7131-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd9ad399-522b-4d12-a7f9-ba38df0c7131" (UID: "cd9ad399-522b-4d12-a7f9-ba38df0c7131"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:52:45.605939 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:45.605896 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ad399-522b-4d12-a7f9-ba38df0c7131-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:46.275784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:46.275749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" event={"ID":"cd9ad399-522b-4d12-a7f9-ba38df0c7131","Type":"ContainerDied","Data":"d521d86390c5ceb374cb9641dafa1f9f07ab2c00e3eed29d6c7b37c0ec96f0ff"} Sep 10 15:52:46.275784 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:46.275776 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz" Sep 10 15:52:46.276038 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:46.275799 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d521d86390c5ceb374cb9641dafa1f9f07ab2c00e3eed29d6c7b37c0ec96f0ff" Sep 10 15:52:47.724371 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.724345 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:52:47.830207 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830178 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-data\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830219 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-certs\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830276 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s75qt\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-kube-api-access-s75qt\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830305 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-podinfo\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830345 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-envoy\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830390 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830379 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-token\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830658 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830403 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-socket\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830658 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830427 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istiod-ca-cert\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830658 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830460 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-credential-socket\") pod \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\" (UID: \"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc\") " Sep 10 15:52:47.830658 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830466 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:52:47.830658 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830557 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:52:47.830910 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830710 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:52:47.830966 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830937 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.831051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830965 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.831051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830980 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.831051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830934 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:52:47.831051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.830960 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:52:47.832645 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.832620 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:52:47.832775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.832652 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-kube-api-access-s75qt" (OuterVolumeSpecName: "kube-api-access-s75qt") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "kube-api-access-s75qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:52:47.832953 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.832930 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:52:47.833062 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.832981 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" (UID: "b9f9e174-6fce-49d7-81c1-690ce6dfcfdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:52:47.932538 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.932500 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75qt\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-kube-api-access-s75qt\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.932538 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.932538 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.932732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.932552 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.932732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.932566 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.932732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.932575 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:47.932732 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:47.932582 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:48.294858 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.294827 2581 generic.go:356] "Generic (PLEG): container finished" podID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerID="5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42" exitCode=0 Sep 10 15:52:48.295046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.294874 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" Sep 10 15:52:48.295046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.294915 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" event={"ID":"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc","Type":"ContainerDied","Data":"5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42"} Sep 10 15:52:48.295046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.294970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr" event={"ID":"b9f9e174-6fce-49d7-81c1-690ce6dfcfdc","Type":"ContainerDied","Data":"0e1c0c7ec2aaddc3d7d14229f81caa6562e2e36ceee5f1fdeee524bfda546724"} Sep 10 15:52:48.295046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.294985 2581 scope.go:117] "RemoveContainer" containerID="5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42" Sep 10 15:52:48.305169 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.305147 2581 scope.go:117] "RemoveContainer" containerID="bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471" Sep 10 15:52:48.313579 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.313563 2581 scope.go:117] "RemoveContainer" containerID="5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42" Sep 10 15:52:48.313833 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:52:48.313815 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42\": container with ID starting with 5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42 not found: ID does not exist" containerID="5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42" Sep 10 15:52:48.313871 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.313844 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42"} err="failed to get container status \"5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42\": rpc error: code = NotFound desc = could not find container \"5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42\": container with ID starting with 5268dc343eae0f72f9617431487c47cbf33a88bd69c600044cee54e061e21d42 not found: ID does not exist" Sep 10 15:52:48.313871 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.313860 2581 scope.go:117] "RemoveContainer" containerID="bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471" Sep 10 15:52:48.314139 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:52:48.314121 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471\": container with ID starting with bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471 not found: ID does not exist" containerID="bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471" Sep 10 15:52:48.314200 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.314143 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471"} err="failed to get container status \"bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471\": rpc error: code = NotFound desc = could not find container \"bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471\": container with ID starting with bf4321b847777a4ba82e2c65e4b3f5e9c81ddf27d30fdc0644f794542422c471 not found: ID does not exist" Sep 10 15:52:48.318369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.318347 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr"] Sep 10 15:52:48.323286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.323261 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1c1464-openshift-default-56494c8866-sqnbr"] Sep 10 15:52:48.788774 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:48.788743 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" path="/var/lib/kubelet/pods/b9f9e174-6fce-49d7-81c1-690ce6dfcfdc/volumes" Sep 10 15:52:58.433827 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.433790 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4"] Sep 10 15:52:58.434342 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.434108 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" podUID="507f2db1-5470-4ddd-a9a6-ab5a92d47e16" containerName="payload-pre-processing" containerID="cri-o://f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c" gracePeriod=30 Sep 10 15:52:58.623091 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.623053 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz"] Sep 10 15:52:58.626976 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.626950 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59c481-dcgnz"] Sep 10 15:52:58.682415 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.682390 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:52:58.744942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.742455 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgc54\" (UniqueName: \"kubernetes.io/projected/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-kube-api-access-jgc54\") pod \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " Sep 10 15:52:58.744942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.742538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-plugins-config-volume\") pod \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\" (UID: \"507f2db1-5470-4ddd-a9a6-ab5a92d47e16\") " Sep 10 15:52:58.744942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.744300 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "507f2db1-5470-4ddd-a9a6-ab5a92d47e16" (UID: "507f2db1-5470-4ddd-a9a6-ab5a92d47e16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:52:58.746889 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.746860 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-kube-api-access-jgc54" (OuterVolumeSpecName: "kube-api-access-jgc54") pod "507f2db1-5470-4ddd-a9a6-ab5a92d47e16" (UID: "507f2db1-5470-4ddd-a9a6-ab5a92d47e16"). InnerVolumeSpecName "kube-api-access-jgc54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:52:58.796365 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.796331 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9ad399-522b-4d12-a7f9-ba38df0c7131" path="/var/lib/kubelet/pods/cd9ad399-522b-4d12-a7f9-ba38df0c7131/volumes" Sep 10 15:52:58.843434 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.843410 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgc54\" (UniqueName: \"kubernetes.io/projected/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-kube-api-access-jgc54\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:58.843434 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:58.843431 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f2db1-5470-4ddd-a9a6-ab5a92d47e16-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:52:59.364721 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.364682 2581 generic.go:356] "Generic (PLEG): container finished" podID="507f2db1-5470-4ddd-a9a6-ab5a92d47e16" containerID="f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c" exitCode=0 Sep 10 15:52:59.364897 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.364750 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" Sep 10 15:52:59.364897 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.364763 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" event={"ID":"507f2db1-5470-4ddd-a9a6-ab5a92d47e16","Type":"ContainerDied","Data":"f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c"} Sep 10 15:52:59.364897 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.364819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4" event={"ID":"507f2db1-5470-4ddd-a9a6-ab5a92d47e16","Type":"ContainerDied","Data":"133875825bfeffa041dc6e21645571d2718dca733e3be20b610710ed475ba5f8"} Sep 10 15:52:59.364897 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.364840 2581 scope.go:117] "RemoveContainer" containerID="f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c" Sep 10 15:52:59.374051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.374032 2581 scope.go:117] "RemoveContainer" containerID="f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c" Sep 10 15:52:59.374328 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:52:59.374308 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c\": container with ID starting with f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c not found: ID does not exist" containerID="f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c" Sep 10 15:52:59.374393 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.374334 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c"} err="failed to get container status \"f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c\": rpc error: code = NotFound desc = could not find container \"f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c\": container with ID starting with f57ab23bd0dd4a2ce5e09d6aedd7860f0248b76614434b46ee646e3a01948f0c not found: ID does not exist" Sep 10 15:52:59.383949 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.383927 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4"] Sep 10 15:52:59.388695 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:52:59.388676 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59c481-5fb669475d-8mxz4"] Sep 10 15:53:00.789905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:00.789875 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="507f2db1-5470-4ddd-a9a6-ab5a92d47e16" path="/var/lib/kubelet/pods/507f2db1-5470-4ddd-a9a6-ab5a92d47e16/volumes" Sep 10 15:53:06.300446 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300408 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245"] Sep 10 15:53:06.300852 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300840 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" Sep 10 15:53:06.300905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300854 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" Sep 10 15:53:06.300905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300878 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507f2db1-5470-4ddd-a9a6-ab5a92d47e16" containerName="payload-pre-processing" Sep 10 15:53:06.300905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300885 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="507f2db1-5470-4ddd-a9a6-ab5a92d47e16" containerName="payload-pre-processing" Sep 10 15:53:06.300905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300898 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9ad399-522b-4d12-a7f9-ba38df0c7131" containerName="revoke-keys" Sep 10 15:53:06.300905 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300904 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9ad399-522b-4d12-a7f9-ba38df0c7131" containerName="revoke-keys" Sep 10 15:53:06.301111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300960 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" Sep 10 15:53:06.301111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300971 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" Sep 10 15:53:06.301111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300978 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="507f2db1-5470-4ddd-a9a6-ab5a92d47e16" containerName="payload-pre-processing" Sep 10 15:53:06.301111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.300985 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9ad399-522b-4d12-a7f9-ba38df0c7131" containerName="revoke-keys" Sep 10 15:53:06.304116 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.304097 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:06.306412 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.306385 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:53:06.306521 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.306440 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:53:06.311497 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.311474 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245"] Sep 10 15:53:06.418275 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.418240 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9851470e-c98f-4c81-a877-7b84e3ddf815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245\" (UID: \"9851470e-c98f-4c81-a877-7b84e3ddf815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:06.518961 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.518924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9851470e-c98f-4c81-a877-7b84e3ddf815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245\" (UID: \"9851470e-c98f-4c81-a877-7b84e3ddf815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:06.519845 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.519817 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9851470e-c98f-4c81-a877-7b84e3ddf815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245\" (UID: \"9851470e-c98f-4c81-a877-7b84e3ddf815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:06.616428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.616347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:06.895295 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:06.895266 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245"] Sep 10 15:53:06.897065 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:53:06.897035 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9851470e_c98f_4c81_a877_7b84e3ddf815.slice/crio-40a6df824327a88da757e22c0e3ed0104819974ded047db7e37f0b1fcd301af9 WatchSource:0}: Error finding container 40a6df824327a88da757e22c0e3ed0104819974ded047db7e37f0b1fcd301af9: Status 404 returned error can't find the container with id 40a6df824327a88da757e22c0e3ed0104819974ded047db7e37f0b1fcd301af9 Sep 10 15:53:07.412675 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:07.412590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" event={"ID":"9851470e-c98f-4c81-a877-7b84e3ddf815","Type":"ContainerStarted","Data":"4ed707fbabc628437ded802c1d446e77c6829a23a6d5d5bf91d9dc071ba4e9f5"} Sep 10 15:53:07.412675 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:07.412647 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" event={"ID":"9851470e-c98f-4c81-a877-7b84e3ddf815","Type":"ContainerStarted","Data":"40a6df824327a88da757e22c0e3ed0104819974ded047db7e37f0b1fcd301af9"} Sep 10 15:53:07.431638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:07.431582 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" podStartSLOduration=1.4315644330000001 podStartE2EDuration="1.431564433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:07.430591318 +0000 UTC m=+1557.213804406" watchObservedRunningTime="2026-09-10 15:53:07.431564433 +0000 UTC m=+1557.214777498" Sep 10 15:53:08.419985 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:08.419950 2581 generic.go:356] "Generic (PLEG): container finished" podID="9851470e-c98f-4c81-a877-7b84e3ddf815" containerID="4ed707fbabc628437ded802c1d446e77c6829a23a6d5d5bf91d9dc071ba4e9f5" exitCode=0 Sep 10 15:53:08.420468 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:08.420027 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" event={"ID":"9851470e-c98f-4c81-a877-7b84e3ddf815","Type":"ContainerDied","Data":"4ed707fbabc628437ded802c1d446e77c6829a23a6d5d5bf91d9dc071ba4e9f5"} Sep 10 15:53:09.570710 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:09.570685 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:09.653978 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:09.653936 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9851470e-c98f-4c81-a877-7b84e3ddf815-maas-api-service-ca\") pod \"9851470e-c98f-4c81-a877-7b84e3ddf815\" (UID: \"9851470e-c98f-4c81-a877-7b84e3ddf815\") " Sep 10 15:53:09.654391 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:09.654363 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9851470e-c98f-4c81-a877-7b84e3ddf815-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9851470e-c98f-4c81-a877-7b84e3ddf815" (UID: "9851470e-c98f-4c81-a877-7b84e3ddf815"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:53:09.755115 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:09.755033 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9851470e-c98f-4c81-a877-7b84e3ddf815-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:53:10.434331 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:10.434291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" event={"ID":"9851470e-c98f-4c81-a877-7b84e3ddf815","Type":"ContainerDied","Data":"40a6df824327a88da757e22c0e3ed0104819974ded047db7e37f0b1fcd301af9"} Sep 10 15:53:10.434331 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:10.434327 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245" Sep 10 15:53:10.434592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:10.434359 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a6df824327a88da757e22c0e3ed0104819974ded047db7e37f0b1fcd301af9" Sep 10 15:53:11.003667 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:11.003636 2581 scope.go:117] "RemoveContainer" containerID="f4c0112f0519b7f87d948645a87fe0a0dfa13d377d05edc96a26e200139bc165" Sep 10 15:53:11.012519 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:11.012498 2581 scope.go:117] "RemoveContainer" containerID="ac17f91fa2f70efd4d787c243bb819e0f862e1a2417dec94b4fd928c6396f61d" Sep 10 15:53:16.895325 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:16.895285 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn"] Sep 10 15:53:16.895746 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:16.895539 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" podUID="3d46a84a-bc32-4c70-a61e-295ef2a148b9" containerName="payload-processing" containerID="cri-o://21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962" gracePeriod=30 Sep 10 15:53:16.903488 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:16.903331 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m"] Sep 10 15:53:16.903599 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:16.903569 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" podUID="8dbde40a-c3f6-4b43-9f9a-12268f61831f" containerName="payload-pre-processing" containerID="cri-o://ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32" gracePeriod=30 Sep 10 15:53:17.123505 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.123469 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245"] Sep 10 15:53:17.134638 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.134604 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-30d70c-rk245"] Sep 10 15:53:17.165556 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.165532 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:53:17.186742 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.186721 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:53:17.332611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.332575 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbde40a-c3f6-4b43-9f9a-12268f61831f-plugins-config-volume\") pod \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " Sep 10 15:53:17.332788 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.332673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46a84a-bc32-4c70-a61e-295ef2a148b9-plugins-config-volume\") pod \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " Sep 10 15:53:17.332788 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.332727 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlpw2\" (UniqueName: \"kubernetes.io/projected/3d46a84a-bc32-4c70-a61e-295ef2a148b9-kube-api-access-zlpw2\") pod \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\" (UID: \"3d46a84a-bc32-4c70-a61e-295ef2a148b9\") " Sep 10 15:53:17.332873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.332785 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7pvv\" (UniqueName: \"kubernetes.io/projected/8dbde40a-c3f6-4b43-9f9a-12268f61831f-kube-api-access-l7pvv\") pod \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\" (UID: \"8dbde40a-c3f6-4b43-9f9a-12268f61831f\") " Sep 10 15:53:17.333069 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.333042 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbde40a-c3f6-4b43-9f9a-12268f61831f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8dbde40a-c3f6-4b43-9f9a-12268f61831f" (UID: "8dbde40a-c3f6-4b43-9f9a-12268f61831f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:53:17.333139 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.333114 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d46a84a-bc32-4c70-a61e-295ef2a148b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d46a84a-bc32-4c70-a61e-295ef2a148b9" (UID: "3d46a84a-bc32-4c70-a61e-295ef2a148b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:53:17.333227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.333212 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbde40a-c3f6-4b43-9f9a-12268f61831f-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:53:17.333283 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.333230 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d46a84a-bc32-4c70-a61e-295ef2a148b9-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:53:17.334830 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.334812 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbde40a-c3f6-4b43-9f9a-12268f61831f-kube-api-access-l7pvv" (OuterVolumeSpecName: "kube-api-access-l7pvv") pod "8dbde40a-c3f6-4b43-9f9a-12268f61831f" (UID: "8dbde40a-c3f6-4b43-9f9a-12268f61831f"). InnerVolumeSpecName "kube-api-access-l7pvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:53:17.334830 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.334822 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d46a84a-bc32-4c70-a61e-295ef2a148b9-kube-api-access-zlpw2" (OuterVolumeSpecName: "kube-api-access-zlpw2") pod "3d46a84a-bc32-4c70-a61e-295ef2a148b9" (UID: "3d46a84a-bc32-4c70-a61e-295ef2a148b9"). InnerVolumeSpecName "kube-api-access-zlpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:53:17.434176 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.434087 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlpw2\" (UniqueName: \"kubernetes.io/projected/3d46a84a-bc32-4c70-a61e-295ef2a148b9-kube-api-access-zlpw2\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:53:17.434176 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.434117 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7pvv\" (UniqueName: \"kubernetes.io/projected/8dbde40a-c3f6-4b43-9f9a-12268f61831f-kube-api-access-l7pvv\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:53:17.473192 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.473154 2581 generic.go:356] "Generic (PLEG): container finished" podID="3d46a84a-bc32-4c70-a61e-295ef2a148b9" containerID="21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962" exitCode=0 Sep 10 15:53:17.473352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.473197 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" Sep 10 15:53:17.473352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.473239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" event={"ID":"3d46a84a-bc32-4c70-a61e-295ef2a148b9","Type":"ContainerDied","Data":"21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962"} Sep 10 15:53:17.473352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.473289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn" event={"ID":"3d46a84a-bc32-4c70-a61e-295ef2a148b9","Type":"ContainerDied","Data":"03be715ec6968ea1f97154171eeaa752855af5f4918a6a504936d64fff76ee80"} Sep 10 15:53:17.473352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.473317 2581 scope.go:117] "RemoveContainer" containerID="21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962" Sep 10 15:53:17.474817 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.474797 2581 generic.go:356] "Generic (PLEG): container finished" podID="8dbde40a-c3f6-4b43-9f9a-12268f61831f" containerID="ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32" exitCode=0 Sep 10 15:53:17.474920 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.474821 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" event={"ID":"8dbde40a-c3f6-4b43-9f9a-12268f61831f","Type":"ContainerDied","Data":"ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32"} Sep 10 15:53:17.474920 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.474843 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" Sep 10 15:53:17.474920 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.474875 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m" event={"ID":"8dbde40a-c3f6-4b43-9f9a-12268f61831f","Type":"ContainerDied","Data":"11f5674aca9c711ca19ded1f47258a248977cb267ad392bc32b92e7c63e0489e"} Sep 10 15:53:17.483322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.483307 2581 scope.go:117] "RemoveContainer" containerID="21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962" Sep 10 15:53:17.483592 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:53:17.483569 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962\": container with ID starting with 21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962 not found: ID does not exist" containerID="21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962" Sep 10 15:53:17.483674 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.483598 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962"} err="failed to get container status \"21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962\": rpc error: code = NotFound desc = could not find container \"21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962\": container with ID starting with 21ff47727851e41f8c168ca62fc15040fd68f09d648bc5ff6164d01350df0962 not found: ID does not exist" Sep 10 15:53:17.483674 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.483614 2581 scope.go:117] "RemoveContainer" containerID="ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32" Sep 10 15:53:17.491708 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.491682 2581 scope.go:117] "RemoveContainer" containerID="ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32" Sep 10 15:53:17.491963 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:53:17.491944 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32\": container with ID starting with ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32 not found: ID does not exist" containerID="ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32" Sep 10 15:53:17.492047 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.491975 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32"} err="failed to get container status \"ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32\": rpc error: code = NotFound desc = could not find container \"ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32\": container with ID starting with ec84b861cbee6281db58afde464273e343f7f136833b05f1920dcfa540c16b32 not found: ID does not exist" Sep 10 15:53:17.497971 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.497946 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn"] Sep 10 15:53:17.500003 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.499979 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30d70c-6965875795-zqfrn"] Sep 10 15:53:17.511730 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.511710 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m"] Sep 10 15:53:17.516065 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:17.516042 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30d70c-7bd5c7ff98-6sv4m"] Sep 10 15:53:18.788512 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:18.788478 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d46a84a-bc32-4c70-a61e-295ef2a148b9" path="/var/lib/kubelet/pods/3d46a84a-bc32-4c70-a61e-295ef2a148b9/volumes" Sep 10 15:53:18.788904 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:18.788854 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dbde40a-c3f6-4b43-9f9a-12268f61831f" path="/var/lib/kubelet/pods/8dbde40a-c3f6-4b43-9f9a-12268f61831f/volumes" Sep 10 15:53:18.789199 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:18.789185 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9851470e-c98f-4c81-a877-7b84e3ddf815" path="/var/lib/kubelet/pods/9851470e-c98f-4c81-a877-7b84e3ddf815/volumes" Sep 10 15:53:24.349031 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.348985 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm"] Sep 10 15:53:24.349424 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349411 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9851470e-c98f-4c81-a877-7b84e3ddf815" containerName="revoke-keys" Sep 10 15:53:24.349424 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349425 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9851470e-c98f-4c81-a877-7b84e3ddf815" containerName="revoke-keys" Sep 10 15:53:24.349501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349436 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" Sep 10 15:53:24.349501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349442 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f9e174-6fce-49d7-81c1-690ce6dfcfdc" containerName="istio-proxy" Sep 10 15:53:24.349501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349455 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbde40a-c3f6-4b43-9f9a-12268f61831f" containerName="payload-pre-processing" Sep 10 15:53:24.349501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349461 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbde40a-c3f6-4b43-9f9a-12268f61831f" containerName="payload-pre-processing" Sep 10 15:53:24.349501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349470 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d46a84a-bc32-4c70-a61e-295ef2a148b9" containerName="payload-processing" Sep 10 15:53:24.349501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349475 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d46a84a-bc32-4c70-a61e-295ef2a148b9" containerName="payload-processing" Sep 10 15:53:24.349686 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349545 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d46a84a-bc32-4c70-a61e-295ef2a148b9" containerName="payload-processing" Sep 10 15:53:24.349686 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349552 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9851470e-c98f-4c81-a877-7b84e3ddf815" containerName="revoke-keys" Sep 10 15:53:24.349686 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.349560 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbde40a-c3f6-4b43-9f9a-12268f61831f" containerName="payload-pre-processing" Sep 10 15:53:24.352592 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.352573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:24.354820 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.354797 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:53:24.354915 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.354800 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:53:24.359944 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.359885 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm"] Sep 10 15:53:24.394886 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.394852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c40d69-235c-4e01-8b39-ddc19651de89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm\" (UID: \"f8c40d69-235c-4e01-8b39-ddc19651de89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:24.496461 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.496400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c40d69-235c-4e01-8b39-ddc19651de89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm\" (UID: \"f8c40d69-235c-4e01-8b39-ddc19651de89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:24.497067 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.497048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c40d69-235c-4e01-8b39-ddc19651de89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm\" (UID: \"f8c40d69-235c-4e01-8b39-ddc19651de89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:24.663602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.663519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:24.944045 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:24.943924 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm"] Sep 10 15:53:24.946291 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:53:24.946264 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c40d69_235c_4e01_8b39_ddc19651de89.slice/crio-662aef9303741b7d4a62cf53f0aa74522a271e156243dba6be8bf4929f2abbad WatchSource:0}: Error finding container 662aef9303741b7d4a62cf53f0aa74522a271e156243dba6be8bf4929f2abbad: Status 404 returned error can't find the container with id 662aef9303741b7d4a62cf53f0aa74522a271e156243dba6be8bf4929f2abbad Sep 10 15:53:25.520741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:25.520654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" event={"ID":"f8c40d69-235c-4e01-8b39-ddc19651de89","Type":"ContainerStarted","Data":"3d2a899ce2a0565d1d7ff74f99ab68f0c80cd8ba5447fa8b6fa097ff54c32a58"} Sep 10 15:53:25.520741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:25.520705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" event={"ID":"f8c40d69-235c-4e01-8b39-ddc19651de89","Type":"ContainerStarted","Data":"662aef9303741b7d4a62cf53f0aa74522a271e156243dba6be8bf4929f2abbad"} Sep 10 15:53:25.537217 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:25.537169 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" podStartSLOduration=1.537154006 podStartE2EDuration="1.537154006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:25.535443485 +0000 UTC m=+1575.318656549" watchObservedRunningTime="2026-09-10 15:53:25.537154006 +0000 UTC m=+1575.320367069" Sep 10 15:53:26.527749 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:26.527714 2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c40d69-235c-4e01-8b39-ddc19651de89" containerID="3d2a899ce2a0565d1d7ff74f99ab68f0c80cd8ba5447fa8b6fa097ff54c32a58" exitCode=0 Sep 10 15:53:26.528132 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:26.527802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" event={"ID":"f8c40d69-235c-4e01-8b39-ddc19651de89","Type":"ContainerDied","Data":"3d2a899ce2a0565d1d7ff74f99ab68f0c80cd8ba5447fa8b6fa097ff54c32a58"} Sep 10 15:53:27.670987 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:27.670954 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:27.725097 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:27.725055 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c40d69-235c-4e01-8b39-ddc19651de89-maas-api-service-ca\") pod \"f8c40d69-235c-4e01-8b39-ddc19651de89\" (UID: \"f8c40d69-235c-4e01-8b39-ddc19651de89\") " Sep 10 15:53:27.725403 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:27.725381 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c40d69-235c-4e01-8b39-ddc19651de89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8c40d69-235c-4e01-8b39-ddc19651de89" (UID: "f8c40d69-235c-4e01-8b39-ddc19651de89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:53:27.825914 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:27.825884 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c40d69-235c-4e01-8b39-ddc19651de89-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:53:28.540816 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:28.540787 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" Sep 10 15:53:28.541010 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:28.540823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm" event={"ID":"f8c40d69-235c-4e01-8b39-ddc19651de89","Type":"ContainerDied","Data":"662aef9303741b7d4a62cf53f0aa74522a271e156243dba6be8bf4929f2abbad"} Sep 10 15:53:28.541010 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:28.540870 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="662aef9303741b7d4a62cf53f0aa74522a271e156243dba6be8bf4929f2abbad" Sep 10 15:53:29.223780 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.223743 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb"] Sep 10 15:53:29.224426 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.224407 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c40d69-235c-4e01-8b39-ddc19651de89" containerName="revoke-keys" Sep 10 15:53:29.224501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.224428 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c40d69-235c-4e01-8b39-ddc19651de89" containerName="revoke-keys" Sep 10 15:53:29.224562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.224528 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c40d69-235c-4e01-8b39-ddc19651de89" containerName="revoke-keys" Sep 10 15:53:29.227130 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.227103 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.230147 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.230127 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8cb756-openshift-default-dockercfg-gggh7\"" Sep 10 15:53:29.242755 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.242729 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb"] Sep 10 15:53:29.339687 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.339626 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2xr\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-kube-api-access-hx2xr\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.339687 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.339687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-envoy\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.339956 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.339725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-data\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.339956 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.339897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-credential-socket\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.340116 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.340026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-podinfo\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.340116 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.340104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istiod-ca-cert\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.340212 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.340150 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-certs\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.340212 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.340179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-socket\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.340293 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.340268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-token\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.440846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.440811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-certs\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.440857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-socket\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.440891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-token\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.440919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2xr\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-kube-api-access-hx2xr\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.440945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-envoy\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.440973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-data\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-credential-socket\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-podinfo\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istiod-ca-cert\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-socket\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441352 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-certs\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441540 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441481 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-data\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441608 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441591 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-credential-socket\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.441885 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.441864 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istiod-ca-cert\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.443342 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.443322 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-envoy\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.443541 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.443521 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-podinfo\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.450909 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.450887 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-token\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.451242 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.451220 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2xr\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-kube-api-access-hx2xr\") pod \"e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.537297 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.537205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:29.816058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.816034 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb"] Sep 10 15:53:29.818344 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:53:29.818309 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a488f5a_bbab_4c78_a597_e22eee64f6c7.slice/crio-f1a70901f1bc6b3a559609e9c80ad8210af07f7278016185a9b493b133238c3e WatchSource:0}: Error finding container f1a70901f1bc6b3a559609e9c80ad8210af07f7278016185a9b493b133238c3e: Status 404 returned error can't find the container with id f1a70901f1bc6b3a559609e9c80ad8210af07f7278016185a9b493b133238c3e Sep 10 15:53:29.820577 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.820546 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:53:29.820647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.820611 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:53:29.820647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:29.820643 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:53:30.557343 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:30.557310 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" event={"ID":"8a488f5a-bbab-4c78-a597-e22eee64f6c7","Type":"ContainerStarted","Data":"8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a"} Sep 10 15:53:30.557775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:30.557360 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" event={"ID":"8a488f5a-bbab-4c78-a597-e22eee64f6c7","Type":"ContainerStarted","Data":"f1a70901f1bc6b3a559609e9c80ad8210af07f7278016185a9b493b133238c3e"} Sep 10 15:53:30.579318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:30.579265 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podStartSLOduration=1.579249481 podStartE2EDuration="1.579249481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:30.577255146 +0000 UTC m=+1580.360468210" watchObservedRunningTime="2026-09-10 15:53:30.579249481 +0000 UTC m=+1580.362462545" Sep 10 15:53:31.538009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:31.537960 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:31.543089 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:31.543065 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:31.563962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:31.563941 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:31.564661 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:31.564643 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:53:34.689285 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:34.689252 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm"] Sep 10 15:53:34.695206 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:34.695176 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ec08ac-7wtcm"] Sep 10 15:53:34.789764 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:34.789728 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c40d69-235c-4e01-8b39-ddc19651de89" path="/var/lib/kubelet/pods/f8c40d69-235c-4e01-8b39-ddc19651de89/volumes" Sep 10 15:53:35.585723 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.585691 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h"] Sep 10 15:53:35.589428 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.589405 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.591915 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.591892 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8cb756\"" Sep 10 15:53:35.592150 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.592135 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8cb756-dockercfg-dt2jg\"" Sep 10 15:53:35.601705 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.601685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h"] Sep 10 15:53:35.704008 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.703945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dzh\" (UniqueName: \"kubernetes.io/projected/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-kube-api-access-t2dzh\") pod \"payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.704470 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.704052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.804716 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.804679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dzh\" (UniqueName: \"kubernetes.io/projected/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-kube-api-access-t2dzh\") pod \"payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.804896 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.804726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.805388 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.805368 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.813611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.813585 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dzh\" (UniqueName: \"kubernetes.io/projected/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-kube-api-access-t2dzh\") pod \"payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:35.900881 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:35.900788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:36.181357 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:36.181308 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h"] Sep 10 15:53:36.181497 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:53:36.181415 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0df6fb_797f_4b5d_a2b8_10fa6da88eb5.slice/crio-dd4b0a5c4b4ea3736733ab557bd30a7d79839dc986f8a4b9e85fad3c60684c81 WatchSource:0}: Error finding container dd4b0a5c4b4ea3736733ab557bd30a7d79839dc986f8a4b9e85fad3c60684c81: Status 404 returned error can't find the container with id dd4b0a5c4b4ea3736733ab557bd30a7d79839dc986f8a4b9e85fad3c60684c81 Sep 10 15:53:36.594291 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:36.594253 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" event={"ID":"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5","Type":"ContainerStarted","Data":"a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70"} Sep 10 15:53:36.594498 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:36.594314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" event={"ID":"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5","Type":"ContainerStarted","Data":"dd4b0a5c4b4ea3736733ab557bd30a7d79839dc986f8a4b9e85fad3c60684c81"} Sep 10 15:53:36.594498 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:36.594457 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:53:36.614771 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:36.614723 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" podStartSLOduration=1.614707713 podStartE2EDuration="1.614707713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:36.612722196 +0000 UTC m=+1586.395935284" watchObservedRunningTime="2026-09-10 15:53:36.614707713 +0000 UTC m=+1586.397920776" Sep 10 15:53:47.606631 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:53:47.606595 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:54:05.769333 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:05.769301 2581 generic.go:356] "Generic (PLEG): container finished" podID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerID="8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a" exitCode=0 Sep 10 15:54:05.769755 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:05.769383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" event={"ID":"8a488f5a-bbab-4c78-a597-e22eee64f6c7","Type":"ContainerDied","Data":"8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a"} Sep 10 15:54:05.769813 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:05.769799 2581 scope.go:117] "RemoveContainer" containerID="8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a" Sep 10 15:54:05.770892 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:05.770869 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:54:05.770941 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:05.770931 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:54:05.770982 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:05.770962 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 10 15:54:06.777094 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:06.777053 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" event={"ID":"8a488f5a-bbab-4c78-a597-e22eee64f6c7","Type":"ContainerStarted","Data":"cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6"} Sep 10 15:54:07.538191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:07.538153 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:54:07.538408 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:07.538388 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 10 15:54:07.538460 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:07.538432 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 10 15:54:08.537895 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:08.537856 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 10 15:54:08.538313 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:08.537911 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 10 15:54:09.537796 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:09.537759 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 10 15:54:09.537981 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:09.537814 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 10 15:54:10.538257 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:10.538223 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 10 15:54:10.538652 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:10.538278 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 10 15:54:11.538025 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:11.537915 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 10 15:54:11.538025 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:11.537980 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 10 15:54:12.538158 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:12.538113 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Sep 10 15:54:12.538550 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:12.538189 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Sep 10 15:54:13.541833 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:13.541801 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:54:13.542291 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:13.542117 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:54:13.542864 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:13.542848 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:54:31.512298 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:54:31.512260 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shkpf_e1e23a13-c442-42e2-880f-9de40551873b/payload-processing/0.log" Sep 10 15:55:08.338694 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:08.338661 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shkpf_e1e23a13-c442-42e2-880f-9de40551873b/payload-processing/0.log" Sep 10 15:55:11.087310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:11.087280 2581 scope.go:117] "RemoveContainer" containerID="a47b6ee7bc22c69cb0dedf53c30de2af71009709a1012a82c5c3781877827a51" Sep 10 15:55:15.757617 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.757583 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8"] Sep 10 15:55:15.760937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.760920 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:15.763401 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.763379 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e48e16\"" Sep 10 15:55:15.763520 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.763460 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e48e16-dockercfg-whx2f\"" Sep 10 15:55:15.769659 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.769638 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8"] Sep 10 15:55:15.791045 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.791018 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh"] Sep 10 15:55:15.793651 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.793636 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.805350 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.805325 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh"] Sep 10 15:55:15.845430 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.845389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknrm\" (UniqueName: \"kubernetes.io/projected/2152868f-d73e-483f-b3f7-c0b5fccb9c37-kube-api-access-pknrm\") pod \"payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:15.845607 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.845555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/fcdee26c-78ea-4e72-889f-bc989eb5aebd-kube-api-access-9brj9\") pod \"payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.845684 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.845639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152868f-d73e-483f-b3f7-c0b5fccb9c37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:15.845819 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.845787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdee26c-78ea-4e72-889f-bc989eb5aebd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.946530 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.946498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152868f-d73e-483f-b3f7-c0b5fccb9c37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:15.946718 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.946579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdee26c-78ea-4e72-889f-bc989eb5aebd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.946718 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.946630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknrm\" (UniqueName: \"kubernetes.io/projected/2152868f-d73e-483f-b3f7-c0b5fccb9c37-kube-api-access-pknrm\") pod \"payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:15.946846 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.946728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/fcdee26c-78ea-4e72-889f-bc989eb5aebd-kube-api-access-9brj9\") pod \"payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.947322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.947298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152868f-d73e-483f-b3f7-c0b5fccb9c37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:15.947322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.947316 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdee26c-78ea-4e72-889f-bc989eb5aebd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.955699 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.955675 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/fcdee26c-78ea-4e72-889f-bc989eb5aebd-kube-api-access-9brj9\") pod \"payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:15.955818 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:15.955704 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknrm\" (UniqueName: \"kubernetes.io/projected/2152868f-d73e-483f-b3f7-c0b5fccb9c37-kube-api-access-pknrm\") pod \"payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:16.072603 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:16.072570 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:16.104431 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:16.104401 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:16.344817 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:16.344786 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8"] Sep 10 15:55:16.346279 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:55:16.346253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2152868f_d73e_483f_b3f7_c0b5fccb9c37.slice/crio-eefac4677b3efeead15eae5bbf90151593a7628ed0a60311e4e704f191590d6c WatchSource:0}: Error finding container eefac4677b3efeead15eae5bbf90151593a7628ed0a60311e4e704f191590d6c: Status 404 returned error can't find the container with id eefac4677b3efeead15eae5bbf90151593a7628ed0a60311e4e704f191590d6c Sep 10 15:55:16.383913 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:16.383885 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh"] Sep 10 15:55:16.385447 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:55:16.385422 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdee26c_78ea_4e72_889f_bc989eb5aebd.slice/crio-db3ea7745692857499be2f3e07231918876e5f0696dffb198665ab0533ff49c8 WatchSource:0}: Error finding container db3ea7745692857499be2f3e07231918876e5f0696dffb198665ab0533ff49c8: Status 404 returned error can't find the container with id db3ea7745692857499be2f3e07231918876e5f0696dffb198665ab0533ff49c8 Sep 10 15:55:17.182441 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.182405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" event={"ID":"fcdee26c-78ea-4e72-889f-bc989eb5aebd","Type":"ContainerStarted","Data":"d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f"} Sep 10 15:55:17.182911 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.182464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" event={"ID":"fcdee26c-78ea-4e72-889f-bc989eb5aebd","Type":"ContainerStarted","Data":"db3ea7745692857499be2f3e07231918876e5f0696dffb198665ab0533ff49c8"} Sep 10 15:55:17.182911 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.182492 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:17.184202 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.184179 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" event={"ID":"2152868f-d73e-483f-b3f7-c0b5fccb9c37","Type":"ContainerStarted","Data":"f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1"} Sep 10 15:55:17.184310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.184225 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" event={"ID":"2152868f-d73e-483f-b3f7-c0b5fccb9c37","Type":"ContainerStarted","Data":"eefac4677b3efeead15eae5bbf90151593a7628ed0a60311e4e704f191590d6c"} Sep 10 15:55:17.184310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.184263 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:17.200723 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.200670 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" podStartSLOduration=2.200656054 podStartE2EDuration="2.200656054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:17.199214318 +0000 UTC m=+1686.982427396" watchObservedRunningTime="2026-09-10 15:55:17.200656054 +0000 UTC m=+1686.983869112" Sep 10 15:55:17.218331 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:17.218284 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" podStartSLOduration=2.218271383 podStartE2EDuration="2.218271383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:17.216278944 +0000 UTC m=+1686.999492009" watchObservedRunningTime="2026-09-10 15:55:17.218271383 +0000 UTC m=+1687.001484506" Sep 10 15:55:28.192574 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:28.192539 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:28.192975 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:28.192844 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:42.674282 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.674193 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq"] Sep 10 15:55:42.677046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.677024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:42.679387 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.679367 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:55:42.679493 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.679368 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:55:42.687117 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.687094 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq"] Sep 10 15:55:42.709838 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.709810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78e02d-2e71-4688-9bd2-f70a3fef81e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq\" (UID: \"8c78e02d-2e71-4688-9bd2-f70a3fef81e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:42.811191 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.811147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78e02d-2e71-4688-9bd2-f70a3fef81e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq\" (UID: \"8c78e02d-2e71-4688-9bd2-f70a3fef81e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:42.811797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.811772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78e02d-2e71-4688-9bd2-f70a3fef81e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq\" (UID: \"8c78e02d-2e71-4688-9bd2-f70a3fef81e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:42.989328 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:42.989238 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:43.267837 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:43.267806 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq"] Sep 10 15:55:43.270443 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:55:43.270415 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c78e02d_2e71_4688_9bd2_f70a3fef81e7.slice/crio-d05097e93350616b44ffe5038320c9fd662ee6a12dd6c00554221a6cc51dba09 WatchSource:0}: Error finding container d05097e93350616b44ffe5038320c9fd662ee6a12dd6c00554221a6cc51dba09: Status 404 returned error can't find the container with id d05097e93350616b44ffe5038320c9fd662ee6a12dd6c00554221a6cc51dba09 Sep 10 15:55:43.338219 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:43.338179 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" event={"ID":"8c78e02d-2e71-4688-9bd2-f70a3fef81e7","Type":"ContainerStarted","Data":"d05097e93350616b44ffe5038320c9fd662ee6a12dd6c00554221a6cc51dba09"} Sep 10 15:55:44.346647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:44.346554 2581 generic.go:356] "Generic (PLEG): container finished" podID="8c78e02d-2e71-4688-9bd2-f70a3fef81e7" containerID="bbc1c935ada81ecdec5916d1eccbb13c45722d12b281bed4b11ed61b9b97b132" exitCode=0 Sep 10 15:55:44.346647 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:44.346613 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" event={"ID":"8c78e02d-2e71-4688-9bd2-f70a3fef81e7","Type":"ContainerDied","Data":"bbc1c935ada81ecdec5916d1eccbb13c45722d12b281bed4b11ed61b9b97b132"} Sep 10 15:55:45.485298 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:45.485275 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:45.538960 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:45.538931 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78e02d-2e71-4688-9bd2-f70a3fef81e7-maas-api-service-ca\") pod \"8c78e02d-2e71-4688-9bd2-f70a3fef81e7\" (UID: \"8c78e02d-2e71-4688-9bd2-f70a3fef81e7\") " Sep 10 15:55:45.539351 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:45.539330 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c78e02d-2e71-4688-9bd2-f70a3fef81e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c78e02d-2e71-4688-9bd2-f70a3fef81e7" (UID: "8c78e02d-2e71-4688-9bd2-f70a3fef81e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:45.640452 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:45.640373 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78e02d-2e71-4688-9bd2-f70a3fef81e7-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:55:46.359341 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:46.359308 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" Sep 10 15:55:46.359341 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:46.359333 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq" event={"ID":"8c78e02d-2e71-4688-9bd2-f70a3fef81e7","Type":"ContainerDied","Data":"d05097e93350616b44ffe5038320c9fd662ee6a12dd6c00554221a6cc51dba09"} Sep 10 15:55:46.359555 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:46.359380 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05097e93350616b44ffe5038320c9fd662ee6a12dd6c00554221a6cc51dba09" Sep 10 15:55:53.652416 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.652378 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh"] Sep 10 15:55:53.652962 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.652614 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" podUID="fcdee26c-78ea-4e72-889f-bc989eb5aebd" containerName="payload-processing" containerID="cri-o://d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f" gracePeriod=30 Sep 10 15:55:53.661731 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.661685 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8"] Sep 10 15:55:53.662069 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.661964 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" podUID="2152868f-d73e-483f-b3f7-c0b5fccb9c37" containerName="payload-pre-processing" containerID="cri-o://f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1" gracePeriod=30 Sep 10 15:55:53.842517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.842484 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq"] Sep 10 15:55:53.844964 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.844934 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e48e16-ld9gq"] Sep 10 15:55:53.939692 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.939668 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:53.942865 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:53.942846 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:54.029161 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029124 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pknrm\" (UniqueName: \"kubernetes.io/projected/2152868f-d73e-483f-b3f7-c0b5fccb9c37-kube-api-access-pknrm\") pod \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " Sep 10 15:55:54.029350 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029215 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152868f-d73e-483f-b3f7-c0b5fccb9c37-plugins-config-volume\") pod \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\" (UID: \"2152868f-d73e-483f-b3f7-c0b5fccb9c37\") " Sep 10 15:55:54.029350 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029249 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/fcdee26c-78ea-4e72-889f-bc989eb5aebd-kube-api-access-9brj9\") pod \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " Sep 10 15:55:54.029479 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029383 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdee26c-78ea-4e72-889f-bc989eb5aebd-plugins-config-volume\") pod \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\" (UID: \"fcdee26c-78ea-4e72-889f-bc989eb5aebd\") " Sep 10 15:55:54.029611 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029571 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2152868f-d73e-483f-b3f7-c0b5fccb9c37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2152868f-d73e-483f-b3f7-c0b5fccb9c37" (UID: "2152868f-d73e-483f-b3f7-c0b5fccb9c37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:54.029754 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029689 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdee26c-78ea-4e72-889f-bc989eb5aebd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fcdee26c-78ea-4e72-889f-bc989eb5aebd" (UID: "fcdee26c-78ea-4e72-889f-bc989eb5aebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:54.029824 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.029751 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152868f-d73e-483f-b3f7-c0b5fccb9c37-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:55:54.031534 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.031505 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2152868f-d73e-483f-b3f7-c0b5fccb9c37-kube-api-access-pknrm" (OuterVolumeSpecName: "kube-api-access-pknrm") pod "2152868f-d73e-483f-b3f7-c0b5fccb9c37" (UID: "2152868f-d73e-483f-b3f7-c0b5fccb9c37"). InnerVolumeSpecName "kube-api-access-pknrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:54.031648 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.031559 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdee26c-78ea-4e72-889f-bc989eb5aebd-kube-api-access-9brj9" (OuterVolumeSpecName: "kube-api-access-9brj9") pod "fcdee26c-78ea-4e72-889f-bc989eb5aebd" (UID: "fcdee26c-78ea-4e72-889f-bc989eb5aebd"). InnerVolumeSpecName "kube-api-access-9brj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:54.130422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.130386 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdee26c-78ea-4e72-889f-bc989eb5aebd-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:55:54.130422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.130415 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pknrm\" (UniqueName: \"kubernetes.io/projected/2152868f-d73e-483f-b3f7-c0b5fccb9c37-kube-api-access-pknrm\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:55:54.130422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.130425 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/fcdee26c-78ea-4e72-889f-bc989eb5aebd-kube-api-access-9brj9\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:55:54.406091 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.406055 2581 generic.go:356] "Generic (PLEG): container finished" podID="2152868f-d73e-483f-b3f7-c0b5fccb9c37" containerID="f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1" exitCode=0 Sep 10 15:55:54.406285 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.406113 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" Sep 10 15:55:54.406285 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.406140 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" event={"ID":"2152868f-d73e-483f-b3f7-c0b5fccb9c37","Type":"ContainerDied","Data":"f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1"} Sep 10 15:55:54.406285 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.406216 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8" event={"ID":"2152868f-d73e-483f-b3f7-c0b5fccb9c37","Type":"ContainerDied","Data":"eefac4677b3efeead15eae5bbf90151593a7628ed0a60311e4e704f191590d6c"} Sep 10 15:55:54.406285 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.406240 2581 scope.go:117] "RemoveContainer" containerID="f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1" Sep 10 15:55:54.407636 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.407604 2581 generic.go:356] "Generic (PLEG): container finished" podID="fcdee26c-78ea-4e72-889f-bc989eb5aebd" containerID="d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f" exitCode=0 Sep 10 15:55:54.407636 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.407634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" event={"ID":"fcdee26c-78ea-4e72-889f-bc989eb5aebd","Type":"ContainerDied","Data":"d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f"} Sep 10 15:55:54.407797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.407665 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" Sep 10 15:55:54.407797 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.407670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh" event={"ID":"fcdee26c-78ea-4e72-889f-bc989eb5aebd","Type":"ContainerDied","Data":"db3ea7745692857499be2f3e07231918876e5f0696dffb198665ab0533ff49c8"} Sep 10 15:55:54.417626 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.417606 2581 scope.go:117] "RemoveContainer" containerID="f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1" Sep 10 15:55:54.417946 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:55:54.417927 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1\": container with ID starting with f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1 not found: ID does not exist" containerID="f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1" Sep 10 15:55:54.418025 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.417952 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1"} err="failed to get container status \"f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1\": rpc error: code = NotFound desc = could not find container \"f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1\": container with ID starting with f3452953befc26598f6a40fa26593e6613052624a57b04c605edf62b35ea8dc1 not found: ID does not exist" Sep 10 15:55:54.418025 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.417971 2581 scope.go:117] "RemoveContainer" containerID="d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f" Sep 10 15:55:54.426158 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.426097 2581 scope.go:117] "RemoveContainer" containerID="d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f" Sep 10 15:55:54.426392 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:55:54.426374 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f\": container with ID starting with d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f not found: ID does not exist" containerID="d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f" Sep 10 15:55:54.426440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.426399 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f"} err="failed to get container status \"d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f\": rpc error: code = NotFound desc = could not find container \"d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f\": container with ID starting with d37cb92f535228cb9a00a9d77531aee3ae7d7e8c6c799f16541bc033d085695f not found: ID does not exist" Sep 10 15:55:54.430902 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.430877 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8"] Sep 10 15:55:54.434877 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.434854 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e48e16-6879866698-nt6h8"] Sep 10 15:55:54.447325 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.447306 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh"] Sep 10 15:55:54.449595 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.449575 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e48e16-8574c96989-2qgzh"] Sep 10 15:55:54.788751 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.788676 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2152868f-d73e-483f-b3f7-c0b5fccb9c37" path="/var/lib/kubelet/pods/2152868f-d73e-483f-b3f7-c0b5fccb9c37/volumes" Sep 10 15:55:54.789111 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.789009 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c78e02d-2e71-4688-9bd2-f70a3fef81e7" path="/var/lib/kubelet/pods/8c78e02d-2e71-4688-9bd2-f70a3fef81e7/volumes" Sep 10 15:55:54.789339 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:55:54.789326 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcdee26c-78ea-4e72-889f-bc989eb5aebd" path="/var/lib/kubelet/pods/fcdee26c-78ea-4e72-889f-bc989eb5aebd/volumes" Sep 10 15:56:04.685414 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685372 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v"] Sep 10 15:56:04.685818 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685802 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2152868f-d73e-483f-b3f7-c0b5fccb9c37" containerName="payload-pre-processing" Sep 10 15:56:04.685818 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685816 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2152868f-d73e-483f-b3f7-c0b5fccb9c37" containerName="payload-pre-processing" Sep 10 15:56:04.685900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685840 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcdee26c-78ea-4e72-889f-bc989eb5aebd" containerName="payload-processing" Sep 10 15:56:04.685900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685846 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcdee26c-78ea-4e72-889f-bc989eb5aebd" containerName="payload-processing" Sep 10 15:56:04.685900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685853 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c78e02d-2e71-4688-9bd2-f70a3fef81e7" containerName="revoke-keys" Sep 10 15:56:04.685900 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685858 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c78e02d-2e71-4688-9bd2-f70a3fef81e7" containerName="revoke-keys" Sep 10 15:56:04.686051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685925 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcdee26c-78ea-4e72-889f-bc989eb5aebd" containerName="payload-processing" Sep 10 15:56:04.686051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685932 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c78e02d-2e71-4688-9bd2-f70a3fef81e7" containerName="revoke-keys" Sep 10 15:56:04.686051 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.685940 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2152868f-d73e-483f-b3f7-c0b5fccb9c37" containerName="payload-pre-processing" Sep 10 15:56:04.689275 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.689258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:04.691851 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.691830 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:56:04.691965 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.691833 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:56:04.697133 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.697104 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v"] Sep 10 15:56:04.844322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.844278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf530a20-fb27-4c19-901a-5e1c90f62907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v\" (UID: \"cf530a20-fb27-4c19-901a-5e1c90f62907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:04.945240 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.945065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf530a20-fb27-4c19-901a-5e1c90f62907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v\" (UID: \"cf530a20-fb27-4c19-901a-5e1c90f62907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:04.945680 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:04.945655 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf530a20-fb27-4c19-901a-5e1c90f62907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v\" (UID: \"cf530a20-fb27-4c19-901a-5e1c90f62907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:05.001595 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:05.001566 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:05.274477 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:05.274453 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v"] Sep 10 15:56:05.276220 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:56:05.276193 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf530a20_fb27_4c19_901a_5e1c90f62907.slice/crio-bc1b6250581d83f33d5ec8bb93af4fd1d3c8d34ee30232474eccba0ff5d6865b WatchSource:0}: Error finding container bc1b6250581d83f33d5ec8bb93af4fd1d3c8d34ee30232474eccba0ff5d6865b: Status 404 returned error can't find the container with id bc1b6250581d83f33d5ec8bb93af4fd1d3c8d34ee30232474eccba0ff5d6865b Sep 10 15:56:05.470393 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:05.470353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" event={"ID":"cf530a20-fb27-4c19-901a-5e1c90f62907","Type":"ContainerStarted","Data":"bc1b6250581d83f33d5ec8bb93af4fd1d3c8d34ee30232474eccba0ff5d6865b"} Sep 10 15:56:06.477758 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:06.477724 2581 generic.go:356] "Generic (PLEG): container finished" podID="cf530a20-fb27-4c19-901a-5e1c90f62907" containerID="e97559c7cb1d7ce4c8929997fa05bf506fb4b06356cefe84ee238480876f3cf8" exitCode=0 Sep 10 15:56:06.478157 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:06.477794 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" event={"ID":"cf530a20-fb27-4c19-901a-5e1c90f62907","Type":"ContainerDied","Data":"e97559c7cb1d7ce4c8929997fa05bf506fb4b06356cefe84ee238480876f3cf8"} Sep 10 15:56:07.620922 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:07.620898 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:07.773668 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:07.773566 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf530a20-fb27-4c19-901a-5e1c90f62907-maas-api-service-ca\") pod \"cf530a20-fb27-4c19-901a-5e1c90f62907\" (UID: \"cf530a20-fb27-4c19-901a-5e1c90f62907\") " Sep 10 15:56:07.773921 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:07.773899 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf530a20-fb27-4c19-901a-5e1c90f62907-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf530a20-fb27-4c19-901a-5e1c90f62907" (UID: "cf530a20-fb27-4c19-901a-5e1c90f62907"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:07.875320 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:07.875288 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf530a20-fb27-4c19-901a-5e1c90f62907-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:56:08.490410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:08.490376 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" event={"ID":"cf530a20-fb27-4c19-901a-5e1c90f62907","Type":"ContainerDied","Data":"bc1b6250581d83f33d5ec8bb93af4fd1d3c8d34ee30232474eccba0ff5d6865b"} Sep 10 15:56:08.490410 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:08.490393 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v" Sep 10 15:56:08.490629 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:08.490423 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc1b6250581d83f33d5ec8bb93af4fd1d3c8d34ee30232474eccba0ff5d6865b" Sep 10 15:56:20.281244 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.281200 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h"] Sep 10 15:56:20.282022 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.281452 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" podUID="3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" containerName="payload-pre-processing" containerID="cri-o://a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70" gracePeriod=30 Sep 10 15:56:20.469714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.469678 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v"] Sep 10 15:56:20.473517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.473490 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cb756-5z45v"] Sep 10 15:56:20.539273 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.539205 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:56:20.560318 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.560279 2581 generic.go:356] "Generic (PLEG): container finished" podID="3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" containerID="a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70" exitCode=0 Sep 10 15:56:20.560517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.560337 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" Sep 10 15:56:20.560517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.560375 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" event={"ID":"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5","Type":"ContainerDied","Data":"a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70"} Sep 10 15:56:20.560517 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.560458 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h" event={"ID":"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5","Type":"ContainerDied","Data":"dd4b0a5c4b4ea3736733ab557bd30a7d79839dc986f8a4b9e85fad3c60684c81"} Sep 10 15:56:20.561286 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.560493 2581 scope.go:117] "RemoveContainer" containerID="a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70" Sep 10 15:56:20.571634 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.571610 2581 scope.go:117] "RemoveContainer" containerID="a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70" Sep 10 15:56:20.571953 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:56:20.571935 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70\": container with ID starting with a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70 not found: ID does not exist" containerID="a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70" Sep 10 15:56:20.572052 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.571960 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70"} err="failed to get container status \"a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70\": rpc error: code = NotFound desc = could not find container \"a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70\": container with ID starting with a6760b62a700a1cc1ae5b62c390c6b30585c7a95349823958c0c00cbe08c2b70 not found: ID does not exist" Sep 10 15:56:20.601443 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.601409 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2dzh\" (UniqueName: \"kubernetes.io/projected/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-kube-api-access-t2dzh\") pod \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " Sep 10 15:56:20.601631 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.601528 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-plugins-config-volume\") pod \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\" (UID: \"3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5\") " Sep 10 15:56:20.602247 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.602211 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" (UID: "3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:20.603714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.603682 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-kube-api-access-t2dzh" (OuterVolumeSpecName: "kube-api-access-t2dzh") pod "3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" (UID: "3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5"). InnerVolumeSpecName "kube-api-access-t2dzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:20.702562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.702528 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:56:20.702562 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.702556 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dzh\" (UniqueName: \"kubernetes.io/projected/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5-kube-api-access-t2dzh\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:56:20.789046 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.789015 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf530a20-fb27-4c19-901a-5e1c90f62907" path="/var/lib/kubelet/pods/cf530a20-fb27-4c19-901a-5e1c90f62907/volumes" Sep 10 15:56:20.878448 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.878416 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h"] Sep 10 15:56:20.883373 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:20.883348 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cb756-57b665b566-p8q8h"] Sep 10 15:56:22.789029 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:56:22.788980 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" path="/var/lib/kubelet/pods/3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5/volumes" Sep 10 15:57:15.534775 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.534739 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb"] Sep 10 15:57:15.535220 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.535013 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" containerID="cri-o://cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6" gracePeriod=30 Sep 10 15:57:15.902690 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.902651 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx"] Sep 10 15:57:15.903195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.903175 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf530a20-fb27-4c19-901a-5e1c90f62907" containerName="revoke-keys" Sep 10 15:57:15.903195 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.903196 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf530a20-fb27-4c19-901a-5e1c90f62907" containerName="revoke-keys" Sep 10 15:57:15.903429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.903223 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" containerName="payload-pre-processing" Sep 10 15:57:15.903429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.903231 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" containerName="payload-pre-processing" Sep 10 15:57:15.903429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.903346 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf530a20-fb27-4c19-901a-5e1c90f62907" containerName="revoke-keys" Sep 10 15:57:15.903429 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.903360 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0df6fb-797f-4b5d-a2b8-10fa6da88eb5" containerName="payload-pre-processing" Sep 10 15:57:15.906805 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.906784 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:15.909080 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.909060 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:57:15.909180 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.909061 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:57:15.915020 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:15.914978 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx"] Sep 10 15:57:16.052100 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.052060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afc6b3b-b33c-4a13-81eb-55cb68f00e81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx\" (UID: \"4afc6b3b-b33c-4a13-81eb-55cb68f00e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:16.153194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.153114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afc6b3b-b33c-4a13-81eb-55cb68f00e81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx\" (UID: \"4afc6b3b-b33c-4a13-81eb-55cb68f00e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:16.153708 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.153689 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afc6b3b-b33c-4a13-81eb-55cb68f00e81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx\" (UID: \"4afc6b3b-b33c-4a13-81eb-55cb68f00e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:16.218310 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.218282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:16.492612 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.492585 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx"] Sep 10 15:57:16.495581 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:57:16.495547 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afc6b3b_b33c_4a13_81eb_55cb68f00e81.slice/crio-83fbc17ad8464de1b61bfaa7e9dcff208732d10369ed96af44eb7c98d484d8a6 WatchSource:0}: Error finding container 83fbc17ad8464de1b61bfaa7e9dcff208732d10369ed96af44eb7c98d484d8a6: Status 404 returned error can't find the container with id 83fbc17ad8464de1b61bfaa7e9dcff208732d10369ed96af44eb7c98d484d8a6 Sep 10 15:57:16.876501 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.876462 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" event={"ID":"4afc6b3b-b33c-4a13-81eb-55cb68f00e81","Type":"ContainerStarted","Data":"8c203743421702676a02292ff59d6fbca9594b1ee7219cd50b151636e20a442f"} Sep 10 15:57:16.876937 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.876524 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" event={"ID":"4afc6b3b-b33c-4a13-81eb-55cb68f00e81","Type":"ContainerStarted","Data":"83fbc17ad8464de1b61bfaa7e9dcff208732d10369ed96af44eb7c98d484d8a6"} Sep 10 15:57:16.894682 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:16.894635 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" podStartSLOduration=1.894620859 podStartE2EDuration="1.894620859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:16.892663474 +0000 UTC m=+1806.675876537" watchObservedRunningTime="2026-09-10 15:57:16.894620859 +0000 UTC m=+1806.677833923" Sep 10 15:57:17.883494 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:17.883456 2581 generic.go:356] "Generic (PLEG): container finished" podID="4afc6b3b-b33c-4a13-81eb-55cb68f00e81" containerID="8c203743421702676a02292ff59d6fbca9594b1ee7219cd50b151636e20a442f" exitCode=0 Sep 10 15:57:17.883863 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:17.883543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" event={"ID":"4afc6b3b-b33c-4a13-81eb-55cb68f00e81","Type":"ContainerDied","Data":"8c203743421702676a02292ff59d6fbca9594b1ee7219cd50b151636e20a442f"} Sep 10 15:57:19.027546 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.027521 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:19.186625 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.186597 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afc6b3b-b33c-4a13-81eb-55cb68f00e81-maas-api-service-ca\") pod \"4afc6b3b-b33c-4a13-81eb-55cb68f00e81\" (UID: \"4afc6b3b-b33c-4a13-81eb-55cb68f00e81\") " Sep 10 15:57:19.187011 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.186970 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afc6b3b-b33c-4a13-81eb-55cb68f00e81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4afc6b3b-b33c-4a13-81eb-55cb68f00e81" (UID: "4afc6b3b-b33c-4a13-81eb-55cb68f00e81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:19.287490 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.287455 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afc6b3b-b33c-4a13-81eb-55cb68f00e81-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:19.898037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.897986 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" Sep 10 15:57:19.898213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.897989 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx" event={"ID":"4afc6b3b-b33c-4a13-81eb-55cb68f00e81","Type":"ContainerDied","Data":"83fbc17ad8464de1b61bfaa7e9dcff208732d10369ed96af44eb7c98d484d8a6"} Sep 10 15:57:19.898213 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:19.898118 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83fbc17ad8464de1b61bfaa7e9dcff208732d10369ed96af44eb7c98d484d8a6" Sep 10 15:57:20.787710 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.787684 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:57:20.902918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.902839 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-envoy\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.902918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.902881 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-certs\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.902918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.902908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx2xr\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-kube-api-access-hx2xr\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.902927 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-credential-socket\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903051 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-data\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903108 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-podinfo\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903152 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-socket\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903255 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903189 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:20.903535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903277 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-token\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903325 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istiod-ca-cert\") pod \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\" (UID: \"8a488f5a-bbab-4c78-a597-e22eee64f6c7\") " Sep 10 15:57:20.903535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903414 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:20.903535 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903482 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:20.903746 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:20.903945 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.903924 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:20.904117 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.904101 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:20.904210 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.904120 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:20.904210 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.904135 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:20.904210 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.904146 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:20.904210 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.904158 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:20.905460 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.905430 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:57:20.905566 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.905505 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:20.905629 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.905608 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-kube-api-access-hx2xr" (OuterVolumeSpecName: "kube-api-access-hx2xr") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "kube-api-access-hx2xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:20.905731 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.905713 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a488f5a-bbab-4c78-a597-e22eee64f6c7" (UID: "8a488f5a-bbab-4c78-a597-e22eee64f6c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:20.906929 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.906908 2581 generic.go:356] "Generic (PLEG): container finished" podID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerID="cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6" exitCode=0 Sep 10 15:57:20.907037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.906946 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" event={"ID":"8a488f5a-bbab-4c78-a597-e22eee64f6c7","Type":"ContainerDied","Data":"cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6"} Sep 10 15:57:20.907037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.906957 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" Sep 10 15:57:20.907037 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.906986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb" event={"ID":"8a488f5a-bbab-4c78-a597-e22eee64f6c7","Type":"ContainerDied","Data":"f1a70901f1bc6b3a559609e9c80ad8210af07f7278016185a9b493b133238c3e"} Sep 10 15:57:20.907175 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.907025 2581 scope.go:117] "RemoveContainer" containerID="cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6" Sep 10 15:57:20.926676 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.926661 2581 scope.go:117] "RemoveContainer" containerID="8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a" Sep 10 15:57:20.934741 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.934726 2581 scope.go:117] "RemoveContainer" containerID="cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6" Sep 10 15:57:20.934987 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:57:20.934970 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6\": container with ID starting with cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6 not found: ID does not exist" containerID="cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6" Sep 10 15:57:20.935048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.935019 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6"} err="failed to get container status \"cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6\": rpc error: code = NotFound desc = could not find container \"cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6\": container with ID starting with cbc4e5864245b047e4ba8f3e53cea67966f38c8021b9d3eab02f2de4bb36a8e6 not found: ID does not exist" Sep 10 15:57:20.935048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.935039 2581 scope.go:117] "RemoveContainer" containerID="8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a" Sep 10 15:57:20.935286 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:57:20.935269 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a\": container with ID starting with 8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a not found: ID does not exist" containerID="8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a" Sep 10 15:57:20.935322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.935293 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a"} err="failed to get container status \"8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a\": rpc error: code = NotFound desc = could not find container \"8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a\": container with ID starting with 8c8a6c1b55366ee5fdfe7a433d992c169ca81ece97e8fbb1d74998f1a7c2135a not found: ID does not exist" Sep 10 15:57:20.938726 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.938706 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb"] Sep 10 15:57:20.943338 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:20.943317 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8cb756-openshift-default-85447fdfc6-8s2pb"] Sep 10 15:57:21.005486 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:21.005454 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.005486 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:21.005483 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.005486 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:21.005492 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a488f5a-bbab-4c78-a597-e22eee64f6c7-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.005700 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:21.005500 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx2xr\" (UniqueName: \"kubernetes.io/projected/8a488f5a-bbab-4c78-a597-e22eee64f6c7-kube-api-access-hx2xr\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:22.789009 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:22.788969 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" path="/var/lib/kubelet/pods/8a488f5a-bbab-4c78-a597-e22eee64f6c7/volumes" Sep 10 15:57:27.119049 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:27.118977 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx"] Sep 10 15:57:27.122885 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:27.122862 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d929c-wn5sx"] Sep 10 15:57:28.788368 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:28.788336 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afc6b3b-b33c-4a13-81eb-55cb68f00e81" path="/var/lib/kubelet/pods/4afc6b3b-b33c-4a13-81eb-55cb68f00e81/volumes" Sep 10 15:57:45.559614 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.559573 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2"] Sep 10 15:57:45.560240 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.559803 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" podUID="4cd2033b-1347-4acb-9831-e5401337ce32" containerName="payload-processing" containerID="cri-o://8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba" gracePeriod=30 Sep 10 15:57:45.823873 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.823810 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:57:45.946504 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.946468 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkh2q\" (UniqueName: \"kubernetes.io/projected/4cd2033b-1347-4acb-9831-e5401337ce32-kube-api-access-wkh2q\") pod \"4cd2033b-1347-4acb-9831-e5401337ce32\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " Sep 10 15:57:45.946713 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.946525 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd2033b-1347-4acb-9831-e5401337ce32-plugins-config-volume\") pod \"4cd2033b-1347-4acb-9831-e5401337ce32\" (UID: \"4cd2033b-1347-4acb-9831-e5401337ce32\") " Sep 10 15:57:45.946923 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.946897 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd2033b-1347-4acb-9831-e5401337ce32-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cd2033b-1347-4acb-9831-e5401337ce32" (UID: "4cd2033b-1347-4acb-9831-e5401337ce32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:45.948667 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:45.948643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd2033b-1347-4acb-9831-e5401337ce32-kube-api-access-wkh2q" (OuterVolumeSpecName: "kube-api-access-wkh2q") pod "4cd2033b-1347-4acb-9831-e5401337ce32" (UID: "4cd2033b-1347-4acb-9831-e5401337ce32"). InnerVolumeSpecName "kube-api-access-wkh2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:46.047340 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.047303 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkh2q\" (UniqueName: \"kubernetes.io/projected/4cd2033b-1347-4acb-9831-e5401337ce32-kube-api-access-wkh2q\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:46.047340 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.047328 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd2033b-1347-4acb-9831-e5401337ce32-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:46.051744 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.051716 2581 generic.go:356] "Generic (PLEG): container finished" podID="4cd2033b-1347-4acb-9831-e5401337ce32" containerID="8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba" exitCode=0 Sep 10 15:57:46.051869 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.051761 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" Sep 10 15:57:46.051869 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.051799 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" event={"ID":"4cd2033b-1347-4acb-9831-e5401337ce32","Type":"ContainerDied","Data":"8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba"} Sep 10 15:57:46.051869 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.051850 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2" event={"ID":"4cd2033b-1347-4acb-9831-e5401337ce32","Type":"ContainerDied","Data":"591f879734509a0e7cd067eb3fcccc5dba59ccfc9bb9014dcd17143604460c8b"} Sep 10 15:57:46.051978 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.051875 2581 scope.go:117] "RemoveContainer" containerID="8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba" Sep 10 15:57:46.060942 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.060926 2581 scope.go:117] "RemoveContainer" containerID="8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba" Sep 10 15:57:46.061228 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:57:46.061208 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba\": container with ID starting with 8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba not found: ID does not exist" containerID="8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba" Sep 10 15:57:46.061295 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.061234 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba"} err="failed to get container status \"8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba\": rpc error: code = NotFound desc = could not find container \"8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba\": container with ID starting with 8e87e504bb72bf10ecb78a81fe0b7d965e3054ae8571d155137add717c492eba not found: ID does not exist" Sep 10 15:57:46.083024 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.082953 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2"] Sep 10 15:57:46.086248 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.086228 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c7257b-669577bb67-2l7q2"] Sep 10 15:57:46.789040 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:46.788980 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd2033b-1347-4acb-9831-e5401337ce32" path="/var/lib/kubelet/pods/4cd2033b-1347-4acb-9831-e5401337ce32/volumes" Sep 10 15:57:53.293697 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.293662 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4"] Sep 10 15:57:53.294206 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.293895 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" containerID="cri-o://f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997" gracePeriod=30 Sep 10 15:57:53.672476 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.672434 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb"] Sep 10 15:57:53.672929 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.672911 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.672932 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.672954 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afc6b3b-b33c-4a13-81eb-55cb68f00e81" containerName="revoke-keys" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.672963 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afc6b3b-b33c-4a13-81eb-55cb68f00e81" containerName="revoke-keys" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.672985 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd2033b-1347-4acb-9831-e5401337ce32" containerName="payload-processing" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673008 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd2033b-1347-4acb-9831-e5401337ce32" containerName="payload-processing" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673035 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" Sep 10 15:57:53.673058 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673043 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" Sep 10 15:57:53.673418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673147 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afc6b3b-b33c-4a13-81eb-55cb68f00e81" containerName="revoke-keys" Sep 10 15:57:53.673418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673159 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" Sep 10 15:57:53.673418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673170 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a488f5a-bbab-4c78-a597-e22eee64f6c7" containerName="istio-proxy" Sep 10 15:57:53.673418 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.673182 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd2033b-1347-4acb-9831-e5401337ce32" containerName="payload-processing" Sep 10 15:57:53.677673 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.677652 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:53.680311 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.680275 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:57:53.680560 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.680538 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 15:57:53.683330 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.683305 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb"] Sep 10 15:57:53.825145 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.825113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d21ad-03d5-4207-bce3-aee25fce28b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb\" (UID: \"093d21ad-03d5-4207-bce3-aee25fce28b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:53.926737 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.926633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d21ad-03d5-4207-bce3-aee25fce28b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb\" (UID: \"093d21ad-03d5-4207-bce3-aee25fce28b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:53.927349 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.927322 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d21ad-03d5-4207-bce3-aee25fce28b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb\" (UID: \"093d21ad-03d5-4207-bce3-aee25fce28b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:53.990118 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:53.990074 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:54.265570 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:54.265543 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb"] Sep 10 15:57:54.267279 ip-10-0-141-82 kubenswrapper[2581]: W0910 15:57:54.267253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093d21ad_03d5_4207_bce3_aee25fce28b4.slice/crio-3ba85bcb0fd5815365c763ba881b5ddafb2486e317440332876deb42ac6a4533 WatchSource:0}: Error finding container 3ba85bcb0fd5815365c763ba881b5ddafb2486e317440332876deb42ac6a4533: Status 404 returned error can't find the container with id 3ba85bcb0fd5815365c763ba881b5ddafb2486e317440332876deb42ac6a4533 Sep 10 15:57:55.105543 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:55.105499 2581 generic.go:356] "Generic (PLEG): container finished" podID="093d21ad-03d5-4207-bce3-aee25fce28b4" containerID="e221e1ff57a92dca14d866f203c990ee2f90fbeaf213f6f7fa15fed7fa57e07f" exitCode=0 Sep 10 15:57:55.105893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:55.105581 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" event={"ID":"093d21ad-03d5-4207-bce3-aee25fce28b4","Type":"ContainerDied","Data":"e221e1ff57a92dca14d866f203c990ee2f90fbeaf213f6f7fa15fed7fa57e07f"} Sep 10 15:57:55.105893 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:55.105634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" event={"ID":"093d21ad-03d5-4207-bce3-aee25fce28b4","Type":"ContainerStarted","Data":"3ba85bcb0fd5815365c763ba881b5ddafb2486e317440332876deb42ac6a4533"} Sep 10 15:57:56.247262 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:56.247236 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:56.352469 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:56.352433 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d21ad-03d5-4207-bce3-aee25fce28b4-maas-api-service-ca\") pod \"093d21ad-03d5-4207-bce3-aee25fce28b4\" (UID: \"093d21ad-03d5-4207-bce3-aee25fce28b4\") " Sep 10 15:57:56.352777 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:56.352755 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093d21ad-03d5-4207-bce3-aee25fce28b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "093d21ad-03d5-4207-bce3-aee25fce28b4" (UID: "093d21ad-03d5-4207-bce3-aee25fce28b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:56.454225 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:56.454139 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093d21ad-03d5-4207-bce3-aee25fce28b4-maas-api-service-ca\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:57.119063 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:57.118969 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" Sep 10 15:57:57.119063 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:57.119007 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb" event={"ID":"093d21ad-03d5-4207-bce3-aee25fce28b4","Type":"ContainerDied","Data":"3ba85bcb0fd5815365c763ba881b5ddafb2486e317440332876deb42ac6a4533"} Sep 10 15:57:57.119063 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:57.119054 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba85bcb0fd5815365c763ba881b5ddafb2486e317440332876deb42ac6a4533" Sep 10 15:57:58.552236 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.552210 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:57:58.676828 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.676737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-podinfo\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.676828 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.676811 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-certs\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.676897 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-envoy\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.676954 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-token\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677048 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.676982 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-socket\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677075 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m9sv\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-kube-api-access-6m9sv\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677125 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istiod-ca-cert\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677147 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:58.677194 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677151 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-credential-socket\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677406 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677213 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-data\") pod \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\" (UID: \"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe\") " Sep 10 15:57:58.677406 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677321 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:58.677406 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677330 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:58.677568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677531 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-data" (OuterVolumeSpecName: "istio-data") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:58.677568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677550 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.677568 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677565 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-credential-socket\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.677714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677580 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-workload-certs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.677714 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.677614 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:58.679039 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.679012 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:57:58.679215 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.679187 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-token" (OuterVolumeSpecName: "istio-token") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:58.679422 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.679395 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-kube-api-access-6m9sv" (OuterVolumeSpecName: "kube-api-access-6m9sv") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "kube-api-access-6m9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:58.679499 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.679471 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" (UID: "e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:57:58.778369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.778331 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-token\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.778369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.778363 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9sv\" (UniqueName: \"kubernetes.io/projected/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-kube-api-access-6m9sv\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.778369 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.778375 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istiod-ca-cert\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.778616 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.778387 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-data\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.778616 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.778401 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-podinfo\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:58.778616 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:58.778411 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe-istio-envoy\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:57:59.133130 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.133096 2581 generic.go:356] "Generic (PLEG): container finished" podID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerID="f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997" exitCode=0 Sep 10 15:57:59.133337 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.133164 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" Sep 10 15:57:59.133337 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.133182 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" event={"ID":"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe","Type":"ContainerDied","Data":"f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997"} Sep 10 15:57:59.133337 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.133240 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4" event={"ID":"e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe","Type":"ContainerDied","Data":"6444099d8f80d463405d38777a4f0ab032d7809e31dad9b6602495fdb58dd193"} Sep 10 15:57:59.133337 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.133263 2581 scope.go:117] "RemoveContainer" containerID="f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997" Sep 10 15:57:59.142301 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.142285 2581 scope.go:117] "RemoveContainer" containerID="7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52" Sep 10 15:57:59.150548 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.150528 2581 scope.go:117] "RemoveContainer" containerID="f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997" Sep 10 15:57:59.150809 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:57:59.150791 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997\": container with ID starting with f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997 not found: ID does not exist" containerID="f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997" Sep 10 15:57:59.150889 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.150816 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997"} err="failed to get container status \"f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997\": rpc error: code = NotFound desc = could not find container \"f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997\": container with ID starting with f948c0dbf09eed2c1038fc3b0c9f8455d1f73a483ef5104da0063c96fa566997 not found: ID does not exist" Sep 10 15:57:59.150889 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.150832 2581 scope.go:117] "RemoveContainer" containerID="7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52" Sep 10 15:57:59.151098 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:57:59.151080 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52\": container with ID starting with 7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52 not found: ID does not exist" containerID="7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52" Sep 10 15:57:59.151145 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.151108 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52"} err="failed to get container status \"7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52\": rpc error: code = NotFound desc = could not find container \"7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52\": container with ID starting with 7cf23b01c20aa9bff39bea8d7e3717ee0227c10851e07fd37c71f8d350e37a52 not found: ID does not exist" Sep 10 15:57:59.156462 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.156439 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4"] Sep 10 15:57:59.161149 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:57:59.161128 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c7257b-openshift-default-745d5d9b44-scgv4"] Sep 10 15:58:00.790244 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:00.790206 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" path="/var/lib/kubelet/pods/e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe/volumes" Sep 10 15:58:04.248021 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.247968 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh"] Sep 10 15:58:04.248440 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.248277 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" podUID="c7d071f4-6d6e-4052-912e-901053f6d5bf" containerName="payload-processing" containerID="cri-o://08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d" gracePeriod=30 Sep 10 15:58:04.433602 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.433566 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb"] Sep 10 15:58:04.436506 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.436475 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3650b0-ldhjb"] Sep 10 15:58:04.514499 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.514430 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:58:04.534750 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.534719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d071f4-6d6e-4052-912e-901053f6d5bf-plugins-config-volume\") pod \"c7d071f4-6d6e-4052-912e-901053f6d5bf\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " Sep 10 15:58:04.534947 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.534854 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pcrs\" (UniqueName: \"kubernetes.io/projected/c7d071f4-6d6e-4052-912e-901053f6d5bf-kube-api-access-9pcrs\") pod \"c7d071f4-6d6e-4052-912e-901053f6d5bf\" (UID: \"c7d071f4-6d6e-4052-912e-901053f6d5bf\") " Sep 10 15:58:04.535152 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.535130 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d071f4-6d6e-4052-912e-901053f6d5bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7d071f4-6d6e-4052-912e-901053f6d5bf" (UID: "c7d071f4-6d6e-4052-912e-901053f6d5bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:58:04.537371 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.537344 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d071f4-6d6e-4052-912e-901053f6d5bf-kube-api-access-9pcrs" (OuterVolumeSpecName: "kube-api-access-9pcrs") pod "c7d071f4-6d6e-4052-912e-901053f6d5bf" (UID: "c7d071f4-6d6e-4052-912e-901053f6d5bf"). InnerVolumeSpecName "kube-api-access-9pcrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:58:04.635918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.635883 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d071f4-6d6e-4052-912e-901053f6d5bf-plugins-config-volume\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:58:04.635918 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.635912 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcrs\" (UniqueName: \"kubernetes.io/projected/c7d071f4-6d6e-4052-912e-901053f6d5bf-kube-api-access-9pcrs\") on node \"ip-10-0-141-82.ec2.internal\" DevicePath \"\"" Sep 10 15:58:04.788317 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:04.788214 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093d21ad-03d5-4207-bce3-aee25fce28b4" path="/var/lib/kubelet/pods/093d21ad-03d5-4207-bce3-aee25fce28b4/volumes" Sep 10 15:58:05.168129 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.168090 2581 generic.go:356] "Generic (PLEG): container finished" podID="c7d071f4-6d6e-4052-912e-901053f6d5bf" containerID="08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d" exitCode=0 Sep 10 15:58:05.168322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.168145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" event={"ID":"c7d071f4-6d6e-4052-912e-901053f6d5bf","Type":"ContainerDied","Data":"08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d"} Sep 10 15:58:05.168322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.168155 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" Sep 10 15:58:05.168322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.168250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh" event={"ID":"c7d071f4-6d6e-4052-912e-901053f6d5bf","Type":"ContainerDied","Data":"6565ade27f14e464f0e572a9fb1283391dfbfb48eb189e8e55670a9b7440ff82"} Sep 10 15:58:05.168322 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.168277 2581 scope.go:117] "RemoveContainer" containerID="08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d" Sep 10 15:58:05.177227 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.177204 2581 scope.go:117] "RemoveContainer" containerID="08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d" Sep 10 15:58:05.177527 ip-10-0-141-82 kubenswrapper[2581]: E0910 15:58:05.177507 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d\": container with ID starting with 08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d not found: ID does not exist" containerID="08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d" Sep 10 15:58:05.177570 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.177536 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d"} err="failed to get container status \"08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d\": rpc error: code = NotFound desc = could not find container \"08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d\": container with ID starting with 08003ce503cd57e0e88916a95b1ecb973060019ee63f4e9f369e6f904712c75d not found: ID does not exist" Sep 10 15:58:05.197036 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.196987 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh"] Sep 10 15:58:05.203296 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:05.203271 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3650b0-6d6d4d99cc-779xh"] Sep 10 15:58:06.788464 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:58:06.788431 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d071f4-6d6e-4052-912e-901053f6d5bf" path="/var/lib/kubelet/pods/c7d071f4-6d6e-4052-912e-901053f6d5bf/volumes" Sep 10 15:59:11.162946 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:59:11.162913 2581 scope.go:117] "RemoveContainer" containerID="4ed707fbabc628437ded802c1d446e77c6829a23a6d5d5bf91d9dc071ba4e9f5" Sep 10 15:59:11.171467 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:59:11.171447 2581 scope.go:117] "RemoveContainer" containerID="20f2c9286273c6dda7d235af48aa068be58453b6ded5e3e751c681123429ee0e" Sep 10 15:59:11.179269 ip-10-0-141-82 kubenswrapper[2581]: I0910 15:59:11.179250 2581 scope.go:117] "RemoveContainer" containerID="5c80140faf853e6c327950c22117211ddb2792d24cd60ca8c7290713392fe9c4" Sep 10 16:00:00.138629 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.138593 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6"] Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.138975 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.138987 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139019 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d071f4-6d6e-4052-912e-901053f6d5bf" containerName="payload-processing" Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139028 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d071f4-6d6e-4052-912e-901053f6d5bf" containerName="payload-processing" Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139051 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093d21ad-03d5-4207-bce3-aee25fce28b4" containerName="revoke-keys" Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139057 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="093d21ad-03d5-4207-bce3-aee25fce28b4" containerName="revoke-keys" Sep 10 16:00:00.139118 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139119 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="093d21ad-03d5-4207-bce3-aee25fce28b4" containerName="revoke-keys" Sep 10 16:00:00.139350 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139126 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" Sep 10 16:00:00.139350 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139133 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" Sep 10 16:00:00.139350 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.139139 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d071f4-6d6e-4052-912e-901053f6d5bf" containerName="payload-processing" Sep 10 16:00:00.142174 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.142154 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" Sep 10 16:00:00.144491 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.144470 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\"" Sep 10 16:00:00.147972 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.147949 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6"] Sep 10 16:00:00.152795 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.152776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" Sep 10 16:00:00.424201 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.424177 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6"] Sep 10 16:00:00.426111 ip-10-0-141-82 kubenswrapper[2581]: W0910 16:00:00.426083 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aa544c_4146_473e_877a_4042af44e0dd.slice/crio-61a2c258e87ec92f91a40018e4cb5c4ef20f6ca30103b5adfd728c71f48fb027 WatchSource:0}: Error finding container 61a2c258e87ec92f91a40018e4cb5c4ef20f6ca30103b5adfd728c71f48fb027: Status 404 returned error can't find the container with id 61a2c258e87ec92f91a40018e4cb5c4ef20f6ca30103b5adfd728c71f48fb027 Sep 10 16:00:00.797309 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.797271 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" event={"ID":"a9aa544c-4146-473e-877a-4042af44e0dd","Type":"ContainerStarted","Data":"9e3130fb96bd21f33cbe48743984c5478da3f4ad802870543762849f7b9c2571"} Sep 10 16:00:00.797472 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.797328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" event={"ID":"a9aa544c-4146-473e-877a-4042af44e0dd","Type":"ContainerStarted","Data":"61a2c258e87ec92f91a40018e4cb5c4ef20f6ca30103b5adfd728c71f48fb027"} Sep 10 16:00:00.830282 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:00.830214 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" podStartSLOduration=0.830200085 podStartE2EDuration="830.200085ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:00.830022449 +0000 UTC m=+1970.613235516" watchObservedRunningTime="2026-09-10 16:00:00.830200085 +0000 UTC m=+1970.613413148" Sep 10 16:00:01.803826 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:01.803796 2581 generic.go:356] "Generic (PLEG): container finished" podID="a9aa544c-4146-473e-877a-4042af44e0dd" containerID="9e3130fb96bd21f33cbe48743984c5478da3f4ad802870543762849f7b9c2571" exitCode=0 Sep 10 16:00:01.804235 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:01.803869 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" event={"ID":"a9aa544c-4146-473e-877a-4042af44e0dd","Type":"ContainerDied","Data":"9e3130fb96bd21f33cbe48743984c5478da3f4ad802870543762849f7b9c2571"} Sep 10 16:00:02.940800 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:02.940747 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" Sep 10 16:00:03.815876 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:03.815844 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" Sep 10 16:00:03.816085 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:03.815882 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-7rkl6" event={"ID":"a9aa544c-4146-473e-877a-4042af44e0dd","Type":"ContainerDied","Data":"61a2c258e87ec92f91a40018e4cb5c4ef20f6ca30103b5adfd728c71f48fb027"} Sep 10 16:00:03.816085 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:03.815928 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61a2c258e87ec92f91a40018e4cb5c4ef20f6ca30103b5adfd728c71f48fb027" Sep 10 16:00:11.205437 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:00:11.205407 2581 scope.go:117] "RemoveContainer" containerID="3d2a899ce2a0565d1d7ff74f99ab68f0c80cd8ba5447fa8b6fa097ff54c32a58" Sep 10 16:02:11.234512 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:02:11.234485 2581 scope.go:117] "RemoveContainer" containerID="e97559c7cb1d7ce4c8929997fa05bf506fb4b06356cefe84ee238480876f3cf8" Sep 10 16:02:11.245836 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:02:11.245810 2581 scope.go:117] "RemoveContainer" containerID="bbc1c935ada81ecdec5916d1eccbb13c45722d12b281bed4b11ed61b9b97b132" Sep 10 16:04:11.277048 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:11.276915 2581 scope.go:117] "RemoveContainer" containerID="e221e1ff57a92dca14d866f203c990ee2f90fbeaf213f6f7fa15fed7fa57e07f" Sep 10 16:04:11.285359 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:11.285337 2581 scope.go:117] "RemoveContainer" containerID="8c203743421702676a02292ff59d6fbca9594b1ee7219cd50b151636e20a442f" Sep 10 16:04:24.124649 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:24.124611 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qmzkc_63ce392e-abb9-43c5-a0a9-572456e49780/manager/0.log" Sep 10 16:04:24.492722 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:24.492645 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-zrgn9_0b8f2b89-20ee-4b22-ba54-bc183318dded/copy-manifests/0.log" Sep 10 16:04:24.501463 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:24.501437 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-zrgn9_0b8f2b89-20ee-4b22-ba54-bc183318dded/manager/0.log" Sep 10 16:04:25.231066 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:25.231034 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817600-7rkl6_a9aa544c-4146-473e-877a-4042af44e0dd/cleanup/0.log" Sep 10 16:04:27.300666 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:27.300631 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cznh2q_311509f8-17b2-4e64-ac62-b15e0ba9f044/istio-proxy/0.log" Sep 10 16:04:27.432371 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:27.432337 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzqhv_e1e76894-3dc5-4ad1-bfbd-be7798b903c9/discovery/0.log" Sep 10 16:04:27.788084 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:27.787981 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/1.log" Sep 10 16:04:28.042350 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:28.042229 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shkpf_e1e23a13-c442-42e2-880f-9de40551873b/payload-processing/0.log" Sep 10 16:04:28.168273 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:28.168241 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fd6757d9d-prmxg_16ae4a17-d168-4499-8387-639457cdb1d4/router/0.log" Sep 10 16:04:29.094245 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:29.094208 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-84v4d_a0be88b9-90f5-4d46-8b22-f26ca8489e31/main/0.log" Sep 10 16:04:37.840753 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:37.840678 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4wc8d_2e5fd8d8-fe5b-4123-ae3e-c5a780cab6e1/global-pull-secret-syncer/0.log" Sep 10 16:04:38.038618 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:38.038583 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vwd5v_d7cb1dfd-6226-4ca4-be73-b1396fe39b5e/konnectivity-agent/0.log" Sep 10 16:04:38.162441 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:38.162355 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-82.ec2.internal_6b2e6a28660fe6b5bb6afb57b77aab26/haproxy/0.log" Sep 10 16:04:44.272161 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.272125 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/alertmanager/0.log" Sep 10 16:04:44.296935 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.296906 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/config-reloader/0.log" Sep 10 16:04:44.321086 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.321061 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/kube-rbac-proxy-web/0.log" Sep 10 16:04:44.354657 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.354624 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/kube-rbac-proxy/0.log" Sep 10 16:04:44.380965 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.380937 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/kube-rbac-proxy-metric/0.log" Sep 10 16:04:44.408446 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.408423 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/prom-label-proxy/0.log" Sep 10 16:04:44.435268 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.435245 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51063e07-8d94-4186-a742-b689ca1690e2/init-config-reloader/0.log" Sep 10 16:04:44.605542 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.605502 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dc458cb77-lx6jn_681f98e0-eacc-4c83-baf5-1bc8f4bbf488/metrics-server/0.log" Sep 10 16:04:44.638618 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.638585 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-t2jfr_bd928188-a6b8-40fe-8a34-3497dd8ecdc2/monitoring-plugin/0.log" Sep 10 16:04:44.693325 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.693298 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bk725_38c0535a-7a83-4ab9-b450-cb9e4024ab7d/node-exporter/0.log" Sep 10 16:04:44.726374 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.726347 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bk725_38c0535a-7a83-4ab9-b450-cb9e4024ab7d/kube-rbac-proxy/0.log" Sep 10 16:04:44.759691 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:44.759662 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bk725_38c0535a-7a83-4ab9-b450-cb9e4024ab7d/init-textfile/0.log" Sep 10 16:04:45.054044 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.054013 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dnt8m_62fd9543-387c-4a0e-be77-3662e87014e0/kube-rbac-proxy-main/0.log" Sep 10 16:04:45.083622 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.083583 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dnt8m_62fd9543-387c-4a0e-be77-3662e87014e0/kube-rbac-proxy-self/0.log" Sep 10 16:04:45.123150 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.123120 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dnt8m_62fd9543-387c-4a0e-be77-3662e87014e0/openshift-state-metrics/0.log" Sep 10 16:04:45.461216 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.461177 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-njh9m_71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9/prometheus-operator/0.log" Sep 10 16:04:45.493633 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.493601 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-njh9m_71d60ed7-85e3-4f8a-98e0-2a5aaf5707b9/kube-rbac-proxy/0.log" Sep 10 16:04:45.525477 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.525446 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-nms9x_ea1fe005-48d7-4140-9d69-4e6873fc7c49/prometheus-operator-admission-webhook/0.log" Sep 10 16:04:45.568104 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.568065 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58d8cc56c9-cnnrq_1733fc28-1877-4e80-a9c8-92f3b47a5b8f/telemeter-client/0.log" Sep 10 16:04:45.604129 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.604074 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58d8cc56c9-cnnrq_1733fc28-1877-4e80-a9c8-92f3b47a5b8f/reload/0.log" Sep 10 16:04:45.639099 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.639065 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58d8cc56c9-cnnrq_1733fc28-1877-4e80-a9c8-92f3b47a5b8f/kube-rbac-proxy/0.log" Sep 10 16:04:45.692070 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.692044 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/thanos-query/0.log" Sep 10 16:04:45.720506 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.720428 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy-web/0.log" Sep 10 16:04:45.749842 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.749812 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy/0.log" Sep 10 16:04:45.791550 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.791519 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/prom-label-proxy/0.log" Sep 10 16:04:45.820152 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.820129 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy-rules/0.log" Sep 10 16:04:45.851187 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:45.851161 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d5bdb4b6-gj4tw_ec489482-f04f-4991-9443-4b64fda06fe2/kube-rbac-proxy-metrics/0.log" Sep 10 16:04:47.244194 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:47.244162 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-ch5hg_a3665a98-3024-4e1a-bf29-ac220d83c587/networking-console-plugin/0.log" Sep 10 16:04:48.212978 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.212943 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f"] Sep 10 16:04:48.213369 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.213356 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" Sep 10 16:04:48.213415 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.213370 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e377d128-6ae0-4fd3-bb68-3f8ad9f3cdbe" containerName="istio-proxy" Sep 10 16:04:48.213415 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.213383 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9aa544c-4146-473e-877a-4042af44e0dd" containerName="cleanup" Sep 10 16:04:48.213415 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.213389 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9aa544c-4146-473e-877a-4042af44e0dd" containerName="cleanup" Sep 10 16:04:48.213512 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.213448 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9aa544c-4146-473e-877a-4042af44e0dd" containerName="cleanup" Sep 10 16:04:48.216218 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.216198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.218534 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.218512 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cs24j\"/\"openshift-service-ca.crt\"" Sep 10 16:04:48.218640 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.218585 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cs24j\"/\"kube-root-ca.crt\"" Sep 10 16:04:48.219440 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.219419 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cs24j\"/\"default-dockercfg-sr2kk\"" Sep 10 16:04:48.232467 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.232444 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f"] Sep 10 16:04:48.322702 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.322672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-lib-modules\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.322702 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.322704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-podres\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.323151 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.322735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-proc\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.323151 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.322857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w556s\" (UniqueName: \"kubernetes.io/projected/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-kube-api-access-w556s\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.323151 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.322881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-sys\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424051 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-lib-modules\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424228 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-podres\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424228 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-proc\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424228 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424165 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-lib-modules\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424228 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424209 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-podres\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424448 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424224 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-proc\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424448 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w556s\" (UniqueName: \"kubernetes.io/projected/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-kube-api-access-w556s\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424448 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-sys\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.424448 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.424366 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-sys\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.433841 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.433816 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w556s\" (UniqueName: \"kubernetes.io/projected/47c466c9-3cec-439b-9ffa-9f8b6cba6f1d-kube-api-access-w556s\") pod \"perf-node-gather-daemonset-nz56f\" (UID: \"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.525807 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.525714 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:48.611553 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.611525 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7955ffc5d9-rxrpm_191083eb-1f95-4ab1-ae6a-5facd2397e5d/console/0.log" Sep 10 16:04:48.665089 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.665053 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-4w9vm_eac25259-b07b-4e50-af0b-f886e275803e/download-server/0.log" Sep 10 16:04:48.801526 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:48.801505 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f"] Sep 10 16:04:48.803554 ip-10-0-141-82 kubenswrapper[2581]: W0910 16:04:48.803526 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod47c466c9_3cec_439b_9ffa_9f8b6cba6f1d.slice/crio-7998b69bfc435b39edba8a9bbb70072f12715036c60282e5b8de34e46a0c127f WatchSource:0}: Error finding container 7998b69bfc435b39edba8a9bbb70072f12715036c60282e5b8de34e46a0c127f: Status 404 returned error can't find the container with id 7998b69bfc435b39edba8a9bbb70072f12715036c60282e5b8de34e46a0c127f Sep 10 16:04:49.319665 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:49.319634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" event={"ID":"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d","Type":"ContainerStarted","Data":"142613c72fa8559d771963a244882af7e73ec7f558c3fc459d930493b7ac5723"} Sep 10 16:04:49.319840 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:49.319680 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" event={"ID":"47c466c9-3cec-439b-9ffa-9f8b6cba6f1d","Type":"ContainerStarted","Data":"7998b69bfc435b39edba8a9bbb70072f12715036c60282e5b8de34e46a0c127f"} Sep 10 16:04:49.319840 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:49.319775 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:49.337369 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:49.337328 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" podStartSLOduration=1.337313307 podStartE2EDuration="1.337313307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:49.336076329 +0000 UTC m=+2259.119289394" watchObservedRunningTime="2026-09-10 16:04:49.337313307 +0000 UTC m=+2259.120526371" Sep 10 16:04:50.944062 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:50.944031 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wh8rn_956f0ede-4db9-4494-ad43-b99970036e50/dns/0.log" Sep 10 16:04:50.978217 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:50.978175 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wh8rn_956f0ede-4db9-4494-ad43-b99970036e50/kube-rbac-proxy/0.log" Sep 10 16:04:51.057258 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:51.057229 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgfdg_1ea8ed65-1c50-40b1-8a47-ade9dd837594/dns-node-resolver/0.log" Sep 10 16:04:51.718544 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:51.718514 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68cc9cc884-t9l6k_e5c2445d-2f2a-48b9-b5c9-d4a5d9717aa8/registry/0.log" Sep 10 16:04:51.839470 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:51.839439 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5ph8_6cbb049d-4320-4bb3-8f6d-0deda397fd56/node-ca/0.log" Sep 10 16:04:52.985927 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:52.985899 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cznh2q_311509f8-17b2-4e64-ac62-b15e0ba9f044/istio-proxy/0.log" Sep 10 16:04:53.561371 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:53.561340 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzqhv_e1e76894-3dc5-4ad1-bfbd-be7798b903c9/discovery/0.log" Sep 10 16:04:53.680807 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:53.680769 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/0.log" Sep 10 16:04:53.696011 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:53.695970 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/1.log" Sep 10 16:04:53.805612 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:53.805585 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shkpf_e1e23a13-c442-42e2-880f-9de40551873b/payload-processing/0.log" Sep 10 16:04:53.837856 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:53.837783 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fd6757d9d-prmxg_16ae4a17-d168-4499-8387-639457cdb1d4/router/0.log" Sep 10 16:04:54.801983 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:54.801952 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zhxq_3a1ae1ee-a086-4096-86d3-4ec6ef4672cd/serve-healthcheck-canary/0.log" Sep 10 16:04:55.335103 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:55.335073 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-nz56f" Sep 10 16:04:55.449354 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:55.449320 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-59q64_2e1d53ff-0078-41fe-9d4e-66c0629657a1/insights-operator/0.log" Sep 10 16:04:55.452410 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:55.452389 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-59q64_2e1d53ff-0078-41fe-9d4e-66c0629657a1/insights-operator/1.log" Sep 10 16:04:55.699034 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:55.698938 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qmv8m_6911314a-52b9-47c0-be21-d844126dc91b/kube-rbac-proxy/0.log" Sep 10 16:04:55.731156 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:55.731125 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qmv8m_6911314a-52b9-47c0-be21-d844126dc91b/exporter/0.log" Sep 10 16:04:55.762065 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:55.762034 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qmv8m_6911314a-52b9-47c0-be21-d844126dc91b/extractor/0.log" Sep 10 16:04:58.597863 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:58.597832 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cznh2q_311509f8-17b2-4e64-ac62-b15e0ba9f044/istio-proxy/0.log" Sep 10 16:04:58.857405 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:58.857370 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qmzkc_63ce392e-abb9-43c5-a0a9-572456e49780/manager/0.log" Sep 10 16:04:59.108921 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.108828 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-zrgn9_0b8f2b89-20ee-4b22-ba54-bc183318dded/manager/0.log" Sep 10 16:04:59.151302 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.151270 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-zrgn9_0b8f2b89-20ee-4b22-ba54-bc183318dded/copy-manifests/0.log" Sep 10 16:04:59.219951 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.219909 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzqhv_e1e76894-3dc5-4ad1-bfbd-be7798b903c9/discovery/0.log" Sep 10 16:04:59.299809 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.299772 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/0.log" Sep 10 16:04:59.312878 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.312846 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/1.log" Sep 10 16:04:59.399048 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.398946 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shkpf_e1e23a13-c442-42e2-880f-9de40551873b/payload-processing/0.log" Sep 10 16:04:59.417411 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:04:59.417381 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fd6757d9d-prmxg_16ae4a17-d168-4499-8387-639457cdb1d4/router/0.log" Sep 10 16:05:00.634339 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:00.634253 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d87857f9b-xh6rl_1bdf80e8-2b1a-4b50-825c-26537c768f3d/manager/0.log" Sep 10 16:05:00.990938 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:00.990851 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cznh2q_311509f8-17b2-4e64-ac62-b15e0ba9f044/istio-proxy/0.log" Sep 10 16:05:01.597010 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:01.596970 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xzqhv_e1e76894-3dc5-4ad1-bfbd-be7798b903c9/discovery/0.log" Sep 10 16:05:01.781284 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:01.781244 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/0.log" Sep 10 16:05:01.795318 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:01.795284 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-md7kz_42c88c16-0207-4f43-a013-4acfc497f484/istio-proxy/1.log" Sep 10 16:05:02.017110 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:02.017022 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-shkpf_e1e23a13-c442-42e2-880f-9de40551873b/payload-processing/0.log" Sep 10 16:05:02.079507 ip-10-0-141-82 kubenswrapper[2581]: I0910 16:05:02.079472 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fd6757d9d-prmxg_16ae4a17-d168-4499-8387-639457cdb1d4/router/0.log"