Jul 28 17:06:56.416855 ip-10-0-132-138 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 17:06:56.416865 ip-10-0-132-138 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 17:06:56.416873 ip-10-0-132-138 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 17:06:56.417093 ip-10-0-132-138 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 17:07:06.565109 ip-10-0-132-138 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 17:07:06.565125 ip-10-0-132-138 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 863c2116ef5b4031a2f1711bab801beb -- Jul 28 17:08:41.463122 ip-10-0-132-138 systemd[1]: Starting Kubernetes Kubelet... Jul 28 17:08:41.882737 ip-10-0-132-138 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:08:41.882737 ip-10-0-132-138 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 17:08:41.882737 ip-10-0-132-138 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:08:41.882737 ip-10-0-132-138 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 17:08:41.882737 ip-10-0-132-138 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:08:41.884459 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.884380 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 17:08:41.888203 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888190 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:41.888203 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888203 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888207 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888210 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888213 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888216 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888219 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888222 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888225 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888228 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888231 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888240 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888243 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888246 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888249 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888252 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:41.888266 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888255 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888311 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888315 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888319 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888322 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888324 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888327 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888330 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888332 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888335 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888338 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888340 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888343 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888345 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888348 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888350 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888354 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888358 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888360 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888363 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:41.888640 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888366 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888369 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888372 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888374 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888379 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888382 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888385 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888388 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888390 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888393 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888395 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888398 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888400 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888402 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888407 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888409 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888412 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888415 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888418 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:41.889141 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888421 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888424 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888427 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888429 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888432 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888434 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888438 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888440 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888443 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888445 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888448 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888450 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888453 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888456 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888466 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888470 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888472 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888475 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888479 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888481 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:41.889642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888484 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888486 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888489 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888491 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888493 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888496 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888499 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888502 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888505 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888508 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888510 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888514 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888517 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888520 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888523 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888526 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888529 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888531 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888534 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:41.890138 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888537 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888539 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888542 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888544 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.888547 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889266 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889273 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889276 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889279 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889282 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889284 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889287 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889295 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889298 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889301 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889303 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889306 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889308 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889311 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:41.890630 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889313 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889317 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889319 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889322 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889325 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889328 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889330 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889333 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889335 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889338 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889340 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889343 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889347 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889351 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889354 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889357 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889359 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889362 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889365 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889367 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:41.891113 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889370 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889372 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889374 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889377 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889380 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889382 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889392 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889395 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889398 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889401 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889403 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889406 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889409 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889411 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889414 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889418 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889421 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889424 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889427 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:41.891642 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889430 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889433 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889436 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889438 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889441 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889444 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889446 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889449 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889451 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889454 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889456 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889458 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889461 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889464 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889466 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889468 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889471 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889474 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889476 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889479 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:41.892133 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889487 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889490 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889493 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889500 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889502 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889505 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889508 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889511 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889515 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889518 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889520 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889523 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889525 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889528 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889530 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889533 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889536 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889538 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889540 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889543 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:41.892637 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889546 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889548 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889551 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889553 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889556 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889558 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889967 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889973 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889976 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889980 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889993 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.889997 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890001 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890004 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890006 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890010 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890014 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890017 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890023 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:41.893135 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890026 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890029 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890031 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890034 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890036 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890039 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890041 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890044 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890046 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890049 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890052 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890054 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890057 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890059 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890062 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890064 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890067 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890069 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890072 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890074 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:41.893607 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890077 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890079 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890083 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890086 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890089 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890091 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890096 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890105 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890109 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890112 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890115 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890120 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890123 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890127 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890130 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890133 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890136 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890138 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890142 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:41.894103 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890146 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890149 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890152 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890155 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890158 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890161 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890163 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890166 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890168 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890170 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890173 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890175 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890178 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890180 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890183 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890188 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890192 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890194 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890197 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890199 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:41.894585 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890202 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890205 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890207 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890210 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890213 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890216 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890218 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890221 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890223 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890226 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890228 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890231 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890233 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890236 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890238 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890241 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890243 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890246 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890248 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890251 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:41.895088 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890253 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890256 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890258 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890261 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890264 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890266 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890269 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890338 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890346 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890355 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890362 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890368 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890373 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890380 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890385 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890388 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890392 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890396 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890399 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890402 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890405 2572 flags.go:64] FLAG: --cgroup-root="" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890408 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890411 2572 flags.go:64] FLAG: --client-ca-file="" Jul 28 17:08:41.895571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890414 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890417 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890420 2572 flags.go:64] FLAG: --cluster-domain="" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890423 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890426 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890429 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890432 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890436 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890439 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890442 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890445 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890448 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890451 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890454 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890457 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890460 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890465 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890468 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890471 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890474 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890477 2572 flags.go:64] FLAG: --enable-server="true" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890480 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890484 2572 flags.go:64] FLAG: --event-burst="100" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890487 2572 flags.go:64] FLAG: --event-qps="50" Jul 28 17:08:41.896140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890491 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890495 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890498 2572 flags.go:64] FLAG: --eviction-hard="" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890502 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890504 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890507 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890510 2572 flags.go:64] FLAG: --eviction-soft="" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890513 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890516 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890519 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890522 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890525 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890528 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890530 2572 flags.go:64] FLAG: --feature-gates="" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890534 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890537 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890540 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890543 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890546 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890548 2572 flags.go:64] FLAG: --help="false" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890551 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-132-138.ec2.internal" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890554 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890557 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 17:08:41.896739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890560 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890564 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890568 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890570 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890573 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890576 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890578 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890582 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890585 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890588 2572 flags.go:64] FLAG: --kube-reserved="" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890592 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890595 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890598 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890601 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890604 2572 flags.go:64] FLAG: --lock-file="" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890607 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890610 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890613 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890618 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890621 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890623 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890626 2572 flags.go:64] FLAG: --logging-format="text" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890629 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890632 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 17:08:41.897390 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890635 2572 flags.go:64] FLAG: --manifest-url="" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890638 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890643 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890646 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890649 2572 flags.go:64] FLAG: --max-pods="110" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890652 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890655 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890658 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890661 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890665 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890668 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890671 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890677 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890680 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890683 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890686 2572 flags.go:64] FLAG: --pod-cidr="" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890689 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890694 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890697 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890701 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890704 2572 flags.go:64] FLAG: --port="10250" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890707 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890710 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c9cbb897e718107c" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890713 2572 flags.go:64] FLAG: --qos-reserved="" Jul 28 17:08:41.897995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890716 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890718 2572 flags.go:64] FLAG: --register-node="true" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890721 2572 flags.go:64] FLAG: --register-with-taints="" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890724 2572 flags.go:64] FLAG: --registry-burst="10" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890727 2572 flags.go:64] FLAG: --registry-qps="5" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890730 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890732 2572 flags.go:64] FLAG: --reserved-memory="" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890736 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890739 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890741 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890744 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890747 2572 flags.go:64] FLAG: --runonce="false" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890750 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890753 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890755 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890758 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890761 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890764 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890767 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890771 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890774 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890777 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890779 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890782 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890785 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 17:08:41.898566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890788 2572 flags.go:64] FLAG: --system-cgroups="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890790 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890796 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890811 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890814 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890818 2572 flags.go:64] FLAG: --tls-min-version="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890821 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890823 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890827 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890830 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890833 2572 flags.go:64] FLAG: --v="2" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890836 2572 flags.go:64] FLAG: --version="false" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890840 2572 flags.go:64] FLAG: --vmodule="" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890843 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.890847 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890974 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890978 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890981 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890985 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890988 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890991 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890994 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890997 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:41.899506 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.890999 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891002 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891005 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891007 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891010 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891013 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891015 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891018 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891021 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891024 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891026 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891029 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891032 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891035 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891037 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891040 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891042 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891045 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891048 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891051 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891053 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:41.900125 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891055 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891058 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891061 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891063 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891066 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891068 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891071 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891074 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891076 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891079 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891081 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891083 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891087 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891089 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891093 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891097 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891100 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891102 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891105 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:41.900654 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891108 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891110 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891113 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891116 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891118 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891121 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891124 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891127 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891129 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891133 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891136 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891139 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891142 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891145 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891148 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891150 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891153 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891155 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891157 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:41.901202 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891160 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891162 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891165 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891167 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891170 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891172 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891176 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891179 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891181 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891184 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891186 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891189 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891193 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891195 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891198 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891201 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891203 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891206 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891208 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891212 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:41.901664 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891214 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891217 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891219 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891222 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891224 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891227 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891230 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891233 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891237 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891240 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891243 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.891246 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.892329 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.900272 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.900285 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 17:08:41.902167 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.900303 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.900312 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900370 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900376 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900379 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900382 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900386 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900389 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900392 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900395 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900398 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900401 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900404 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900407 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900409 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900412 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900414 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900417 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900419 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:41.902538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900422 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900424 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900427 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900430 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900432 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900435 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900437 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900441 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900445 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900448 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900451 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900454 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900457 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900459 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900462 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900464 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900467 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900470 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900472 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900475 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:41.903007 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900477 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900480 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900482 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900485 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900487 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900490 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900493 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900495 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900498 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900502 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900505 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900508 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900511 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900513 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900516 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900518 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900521 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900523 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900526 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900529 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:41.903490 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900531 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900534 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900536 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900539 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900542 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900544 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900546 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900556 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900560 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900562 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900564 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900567 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900570 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900572 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900575 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900579 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900582 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900585 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900588 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900591 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:41.904068 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900593 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900596 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900605 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900608 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900610 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900613 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900616 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900618 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900621 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900623 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900625 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900628 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900631 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900633 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900636 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900638 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900641 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900643 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900646 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:41.904538 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900649 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900651 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900654 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.900659 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900772 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900778 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900780 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900783 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900785 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900788 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900791 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900793 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900795 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900816 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900821 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:41.905115 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900824 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900833 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900835 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900838 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900840 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900843 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900845 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900848 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900850 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900853 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900856 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900858 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900861 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900864 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900866 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900869 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900871 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900874 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900876 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:41.905478 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900878 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900881 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900884 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900886 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900889 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900891 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900894 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900897 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900899 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900902 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900904 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900907 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900909 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900912 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900914 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900922 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900925 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900928 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900930 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900933 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:41.905954 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900935 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900937 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900940 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900943 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900945 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900948 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900950 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900952 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900955 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900957 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900960 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900963 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900965 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900968 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900970 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900973 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900976 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900978 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900982 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:41.906438 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900985 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900988 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900990 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900993 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900996 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.900998 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901001 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901003 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901006 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901016 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901018 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901021 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901023 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901026 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901029 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901033 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901035 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901038 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901040 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901043 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:41.906920 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901045 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901048 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901050 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901053 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901055 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901057 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901062 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901065 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901068 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:41.901071 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.901075 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.901168 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 17:08:41.907392 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.906690 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 17:08:41.908003 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.907990 2572 server.go:1019] "Starting client certificate rotation" Jul 28 17:08:41.908105 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.908087 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:08:41.908133 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.908127 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:08:41.910729 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.910716 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:08:41.915527 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.915508 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:08:41.929603 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.929585 2572 log.go:25] "Validated CRI v1 runtime API" Jul 28 17:08:41.931574 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.931561 2572 log.go:25] "Validated CRI v1 image API" Jul 28 17:08:41.932285 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.932271 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 17:08:41.936140 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.936122 2572 fs.go:135] Filesystem UUIDs: map[5ad43cfa-9707-4996-ac6f-70595751c0bd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8594d98d-1955-4e13-af2e-8f35c26d47fe:/dev/nvme0n1p3] Jul 28 17:08:41.936666 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.936139 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 17:08:41.941584 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.941558 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:08:41.943530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.943386 2572 manager.go:216] Machine: {Timestamp:2026-07-28 17:08:41.942178597 +0000 UTC m=+0.360539637 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103163 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec235eb715c2530c0710ae448abc96be SystemUUID:ec235eb7-15c2-530c-0710-ae448abc96be BootID:863c2116-ef5b-4031-a2f1-711bab801beb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:70:5d:b3:f9:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:70:5d:b3:f9:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:81:7c:58:9b:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 17:08:41.943530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.943521 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 17:08:41.943655 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.943597 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 17:08:41.946783 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.946758 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 17:08:41.946925 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.946786 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 17:08:41.947554 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.947544 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 17:08:41.947587 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.947557 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 17:08:41.947587 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.947564 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:08:41.948648 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.948637 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:08:41.949358 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.949347 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 17:08:41.950796 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.950785 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:08:41.950924 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.950914 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 17:08:41.950963 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.950955 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 28 17:08:41.950995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.950974 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 17:08:41.950995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.950987 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 17:08:41.951067 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.950995 2572 kubelet.go:398] "Adding apiserver pod source" Jul 28 17:08:41.951067 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.951005 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 17:08:41.951536 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.951522 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:08:41.951588 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.951554 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:08:41.954823 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.954789 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 17:08:41.956506 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.956493 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 17:08:41.956561 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.956523 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 17:08:41.957983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.957971 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.957987 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.957993 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.957999 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958008 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958013 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958021 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958026 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958034 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958039 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958047 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 17:08:41.958057 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958052 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 17:08:41.958897 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958887 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 17:08:41.958897 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.958897 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 17:08:41.962186 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.962172 2572 server.go:1268] "Started kubelet" Jul 28 17:08:41.962268 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.962244 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 17:08:41.962372 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.962265 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 17:08:41.962372 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.962344 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 17:08:41.962972 ip-10-0-132-138 systemd[1]: Started Kubernetes Kubelet. Jul 28 17:08:41.965000 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.964973 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 17:08:41.965152 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.965126 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 17:08:41.965249 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.964979 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 28 17:08:41.965249 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.965161 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 17:08:41.966337 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.966322 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 17:08:41.969225 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.969207 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 17:08:41.969823 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.968654 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-138.ec2.internal.18c6832ac1caef4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-138.ec2.internal,UID:ip-10-0-132-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-138.ec2.internal,},FirstTimestamp:2026-07-28 17:08:41.962147662 +0000 UTC m=+0.380508703,LastTimestamp:2026-07-28 17:08:41.962147662 +0000 UTC m=+0.380508703,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-138.ec2.internal,}" Jul 28 17:08:41.971904 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.971883 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 17:08:41.971904 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.971900 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 17:08:41.972480 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.972069 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:41.972574 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972494 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 17:08:41.972574 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972507 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 17:08:41.972574 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972522 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 17:08:41.972902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972738 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 28 17:08:41.972902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972751 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 28 17:08:41.972902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972852 2572 factory.go:55] Registering systemd factory Jul 28 17:08:41.972902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.972899 2572 factory.go:223] Registration of the systemd container factory successfully Jul 28 17:08:41.973108 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.973086 2572 factory.go:153] Registering CRI-O factory Jul 28 17:08:41.973108 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.973100 2572 factory.go:223] Registration of the crio container factory successfully Jul 28 17:08:41.973177 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.973152 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 17:08:41.973177 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.973175 2572 factory.go:103] Registering Raw factory Jul 28 17:08:41.973250 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.973189 2572 manager.go:1196] Started watching for new ooms in manager Jul 28 17:08:41.973678 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.973665 2572 manager.go:349] Starting recovery of all containers Jul 28 17:08:41.978619 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.978584 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 17:08:41.978734 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.978718 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 17:08:41.984746 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.984728 2572 manager.go:354] Recovery completed Jul 28 17:08:41.989104 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.989091 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:41.990770 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.990753 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:41.990843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.990786 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:41.990843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.990817 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:41.991353 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.991339 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 17:08:41.991353 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.991351 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 17:08:41.991477 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.991367 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:08:41.991477 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.991353 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:41.992414 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.992397 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:41.992499 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.992428 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:41.992499 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.992442 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:41.992928 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.992912 2572 policy_none.go:49] "None policy: Start" Jul 28 17:08:41.993005 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.992932 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 17:08:41.993005 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.992946 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 17:08:41.993318 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:41.993237 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-138.ec2.internal.18c6832ac37fad0d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-138.ec2.internal,UID:ip-10-0-132-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-138.ec2.internal,},FirstTimestamp:2026-07-28 17:08:41.990769933 +0000 UTC m=+0.409130973,LastTimestamp:2026-07-28 17:08:41.990769933 +0000 UTC m=+0.409130973,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-138.ec2.internal,}" Jul 28 17:08:41.993941 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:41.993925 2572 policy_none.go:47] "Start" Jul 28 17:08:42.000915 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.000895 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lmqh8" Jul 28 17:08:42.002638 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.002558 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-138.ec2.internal.18c6832ac38007fb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-138.ec2.internal,UID:ip-10-0-132-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-138.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-138.ec2.internal,},FirstTimestamp:2026-07-28 17:08:41.990793211 +0000 UTC m=+0.409154251,LastTimestamp:2026-07-28 17:08:41.990793211 +0000 UTC m=+0.409154251,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-138.ec2.internal,}" Jul 28 17:08:42.009234 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.009217 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lmqh8" Jul 28 17:08:42.020427 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.020413 2572 manager.go:337] "Starting Device Plugin manager" Jul 28 17:08:42.020501 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.020444 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 17:08:42.020501 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.020457 2572 server.go:87] "Starting device plugin registration server" Jul 28 17:08:42.020729 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.020716 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 17:08:42.020854 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.020729 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 17:08:42.020941 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.020928 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 17:08:42.021008 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.020998 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 17:08:42.021008 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.021006 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 17:08:42.021524 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.021503 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 17:08:42.021614 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.021559 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.038118 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.038103 2572 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2606 into the system container \"/system.slice\": " Jul 28 17:08:42.101868 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.101188 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 17:08:42.102560 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.102545 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 17:08:42.102608 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.102589 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 17:08:42.102652 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.102614 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 17:08:42.102703 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.102656 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 17:08:42.105654 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.105638 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:42.121426 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.121406 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:42.122217 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.122198 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:42.122282 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.122234 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:42.122282 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.122248 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:42.122282 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.122271 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.133231 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.133170 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.133231 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.133191 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-138.ec2.internal\": node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.177684 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.177523 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.203615 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.203592 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal"] Jul 28 17:08:42.203699 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.203649 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:42.204501 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.204484 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:42.204585 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.204512 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:42.204585 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.204522 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:42.205602 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.205590 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:42.205734 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.205720 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.205771 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.205750 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:42.206258 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.206242 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:42.206350 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.206272 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:42.206350 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.206284 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:42.206350 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.206246 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:42.206350 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.206350 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:42.206534 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.206366 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:42.207171 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.207157 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.207237 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.207181 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:42.207822 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.207793 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:42.207916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.207835 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:42.207916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.207849 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:42.230389 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.230373 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-138.ec2.internal\" not found" node="ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.234773 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.234757 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-138.ec2.internal\" not found" node="ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.274241 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.274219 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a28b7e1bb7bab65a1df8ef56c8ebc188-config\") pod \"kube-apiserver-proxy-ip-10-0-132-138.ec2.internal\" (UID: \"a28b7e1bb7bab65a1df8ef56c8ebc188\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.274326 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.274248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76db135eb9a95b201fdaec547a8e38e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal\" (UID: \"76db135eb9a95b201fdaec547a8e38e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.274326 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.274274 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76db135eb9a95b201fdaec547a8e38e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal\" (UID: \"76db135eb9a95b201fdaec547a8e38e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.278312 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.278298 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.374810 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.374773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76db135eb9a95b201fdaec547a8e38e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal\" (UID: \"76db135eb9a95b201fdaec547a8e38e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.374930 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.374825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76db135eb9a95b201fdaec547a8e38e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal\" (UID: \"76db135eb9a95b201fdaec547a8e38e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.374930 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.374843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a28b7e1bb7bab65a1df8ef56c8ebc188-config\") pod \"kube-apiserver-proxy-ip-10-0-132-138.ec2.internal\" (UID: \"a28b7e1bb7bab65a1df8ef56c8ebc188\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.374930 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.374863 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76db135eb9a95b201fdaec547a8e38e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal\" (UID: \"76db135eb9a95b201fdaec547a8e38e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.374930 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.374880 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76db135eb9a95b201fdaec547a8e38e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal\" (UID: \"76db135eb9a95b201fdaec547a8e38e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.374930 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.374868 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a28b7e1bb7bab65a1df8ef56c8ebc188-config\") pod \"kube-apiserver-proxy-ip-10-0-132-138.ec2.internal\" (UID: \"a28b7e1bb7bab65a1df8ef56c8ebc188\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.378884 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.378865 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.479727 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.479639 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.532858 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.532836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.538432 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.538408 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" Jul 28 17:08:42.580335 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.580279 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.680879 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.680839 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.781353 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.781299 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.790857 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.790831 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:42.881858 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:42.881835 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-138.ec2.internal\" not found" Jul 28 17:08:42.888608 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.888590 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:42.907052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.907033 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 17:08:42.907156 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.907138 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:08:42.907205 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.907185 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:08:42.907240 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.907185 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:08:42.951345 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:42.951321 2572 apiserver.go:52] "Watching apiserver" Jul 28 17:08:44.201292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.201272 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 17:08:44.201292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.201286 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" Jul 28 17:08:44.202412 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.202378 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 17:08:44.202703 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.202686 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-m2hqd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr","openshift-cluster-node-tuning-operator/tuned-ckr54","openshift-dns/node-resolver-npp2t","openshift-multus/multus-wm8lw","openshift-multus/network-metrics-daemon-m4kwh","openshift-network-operator/iptables-alerter-thtn7","openshift-ovn-kubernetes/ovnkube-node-rpnss","kube-system/global-pull-secret-syncer-nc9d4","openshift-image-registry/node-ca-tdhg2","openshift-multus/multus-additional-cni-plugins-gljn7","openshift-network-diagnostics/network-check-target-k6847"] Jul 28 17:08:44.203955 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.203940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.204843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.204824 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.205687 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.205667 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.206252 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.206239 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zlxrn\"" Jul 28 17:08:44.206479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.206465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.207425 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.207410 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.208368 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208351 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.208446 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208405 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 17:08:44.208502 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208443 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bm2jz\"" Jul 28 17:08:44.208551 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.208436 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:44.208815 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208795 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.208902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208795 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwbgf\"" Jul 28 17:08:44.208902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208839 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.208902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208849 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.209056 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208944 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.209056 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.208972 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 17:08:44.209185 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209069 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.209185 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209162 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.209285 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209258 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vl2z5\"" Jul 28 17:08:44.209355 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209338 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 17:08:44.209417 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.209494 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209478 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k4c84\"" Jul 28 17:08:44.209537 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209500 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 17:08:44.209672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.209653 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.210608 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.210592 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.211961 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.211925 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.211961 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.211929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.212101 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.212001 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 17:08:44.212101 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.211997 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:44.212855 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.212841 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.212914 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.212844 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.212914 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.212905 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 17:08:44.213309 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.213297 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.213606 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.213589 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.213724 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.213676 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 17:08:44.213826 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.213686 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 17:08:44.213921 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.213676 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jjrll\"" Jul 28 17:08:44.213969 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.213923 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d2869\"" Jul 28 17:08:44.214196 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.214177 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.214800 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.214326 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 17:08:44.214913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.214896 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 17:08:44.215266 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.215251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.215452 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.215432 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 17:08:44.216380 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.216360 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wksfn\"" Jul 28 17:08:44.216485 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.216409 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 17:08:44.216544 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.216536 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 17:08:44.217032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.217012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:44.217337 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.217316 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:44.217458 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.217373 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 17:08:44.217890 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.217873 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 17:08:44.221320 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.221298 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xr86f\"" Jul 28 17:08:44.221429 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.221377 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:08:44.221504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.221467 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" Jul 28 17:08:44.221563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.221533 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:08:44.222175 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.222158 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal"] Jul 28 17:08:44.238982 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.238966 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal"] Jul 28 17:08:44.239072 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.239010 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:08:44.241242 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.241208 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:03:43 +0000 UTC" deadline="2028-04-03 05:25:11.163198328 +0000 UTC" Jul 28 17:08:44.241310 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.241242 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14748h16m26.921959907s" Jul 28 17:08:44.302325 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.302304 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 17:08:44.306702 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306680 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-socket-dir-parent\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.306781 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-daemon-config\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.306781 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/156ff3fc-7c3f-49d5-a362-949d57777cf1-host\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.306781 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-os-release\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.306916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-kube-api-access-xnqlg\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.306916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d1534dc-38b2-4881-9fd1-80795b1e4b24-kubelet-config\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.306916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrqr\" (UniqueName: \"kubernetes.io/projected/156ff3fc-7c3f-49d5-a362-949d57777cf1-kube-api-access-svrqr\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.306916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-cni-bin\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.306916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-tmp-dir\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.307080 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-run-netns\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307080 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/156ff3fc-7c3f-49d5-a362-949d57777cf1-serviceca\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.307080 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.306970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.307080 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307001 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-registration-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.307080 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-sys-fs\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-kubelet\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-node-log\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cnibin\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-system-cni-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.307218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-cni-binary-copy\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmlv\" (UniqueName: \"kubernetes.io/projected/61deeab5-0b0e-4a34-80ea-e476ec574ee7-kube-api-access-5lmlv\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307255 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddbp\" (UniqueName: \"kubernetes.io/projected/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-kube-api-access-cddbp\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-cni-bin\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c921956-e37e-4cb1-920c-bd7736f44fa1-konnectivity-ca\") pod \"konnectivity-agent-m2hqd\" (UID: \"2c921956-e37e-4cb1-920c-bd7736f44fa1\") " pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307359 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-lib-modules\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.307423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-var-lib-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307438 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-device-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-etc-kubernetes\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307489 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307512 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzwl\" (UniqueName: \"kubernetes.io/projected/071211d6-06de-48c3-8f5e-fd7b5edc77d7-kube-api-access-6pzwl\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307546 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysconfig\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-systemd\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-run-ovn-kubernetes\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7940f71f-a825-4461-abb6-4f7f87919491-ovn-node-metrics-cert\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxpp4\" (UniqueName: \"kubernetes.io/projected/7940f71f-a825-4461-abb6-4f7f87919491-kube-api-access-wxpp4\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.307670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-cni-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-netns\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-multus-certs\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307721 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-sys\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307739 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-ovn\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-run\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-host\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307808 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dea6475c-946b-489d-8730-8a4781dfe17c-etc-tuned\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbb89330-d651-44e6-b2cf-2f813c42aa6b-host-slash\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-systemd-units\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307871 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-slash\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-env-overrides\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-etc-selinux\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-kubernetes\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972t2\" (UniqueName: \"kubernetes.io/projected/71902b90-5bb9-481e-9c6a-08cef1630d47-kube-api-access-972t2\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.307983 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-hostroot\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.308208 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308018 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-modprobe-d\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysctl-d\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d1534dc-38b2-4881-9fd1-80795b1e4b24-dbus\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-systemd\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-log-socket\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-ovnkube-config\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308705 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-cni-multus\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-kubelet\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308760 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-conf-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-system-cni-dir\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysctl-conf\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-cnibin\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-os-release\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.309002 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-var-lib-kubelet\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbfd\" (UniqueName: \"kubernetes.io/projected/dea6475c-946b-489d-8730-8a4781dfe17c-kube-api-access-qdbfd\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-hosts-file\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-etc-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308980 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-socket-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.308996 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-k8s-cni-cncf-io\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.309025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea6475c-946b-489d-8730-8a4781dfe17c-tmp\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.309045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbb89330-d651-44e6-b2cf-2f813c42aa6b-iptables-alerter-script\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.309068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtlt\" (UniqueName: \"kubernetes.io/projected/bbb89330-d651-44e6-b2cf-2f813c42aa6b-kube-api-access-6jtlt\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.309093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-cni-netd\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.309113 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-ovnkube-script-lib\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.309533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.309127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c921956-e37e-4cb1-920c-bd7736f44fa1-agent-certs\") pod \"konnectivity-agent-m2hqd\" (UID: \"2c921956-e37e-4cb1-920c-bd7736f44fa1\") " pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.334426 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.334407 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2xvn" Jul 28 17:08:44.345767 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.345748 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2xvn" Jul 28 17:08:44.409310 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-socket-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409314 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-k8s-cni-cncf-io\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea6475c-946b-489d-8730-8a4781dfe17c-tmp\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409352 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbb89330-d651-44e6-b2cf-2f813c42aa6b-iptables-alerter-script\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409367 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtlt\" (UniqueName: \"kubernetes.io/projected/bbb89330-d651-44e6-b2cf-2f813c42aa6b-kube-api-access-6jtlt\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-cni-netd\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409433 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-k8s-cni-cncf-io\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409439 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-cni-netd\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409468 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-socket-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-ovnkube-script-lib\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c921956-e37e-4cb1-920c-bd7736f44fa1-agent-certs\") pod \"konnectivity-agent-m2hqd\" (UID: \"2c921956-e37e-4cb1-920c-bd7736f44fa1\") " pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409548 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-socket-dir-parent\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-daemon-config\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/156ff3fc-7c3f-49d5-a362-949d57777cf1-host\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-os-release\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-kube-api-access-xnqlg\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d1534dc-38b2-4881-9fd1-80795b1e4b24-kubelet-config\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrqr\" (UniqueName: \"kubernetes.io/projected/156ff3fc-7c3f-49d5-a362-949d57777cf1-kube-api-access-svrqr\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409711 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-cni-bin\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-tmp-dir\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409744 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-socket-dir-parent\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.409763 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-run-netns\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409663 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/156ff3fc-7c3f-49d5-a362-949d57777cf1-serviceca\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-registration-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409855 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-sys-fs\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-kubelet\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-node-log\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cnibin\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.409988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-system-cni-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410013 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-cni-binary-copy\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410032 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d1534dc-38b2-4881-9fd1-80795b1e4b24-kubelet-config\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmlv\" (UniqueName: \"kubernetes.io/projected/61deeab5-0b0e-4a34-80ea-e476ec574ee7-kube-api-access-5lmlv\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410073 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-ovnkube-script-lib\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410084 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbb89330-d651-44e6-b2cf-2f813c42aa6b-iptables-alerter-script\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.410402 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410171 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-daemon-config\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410183 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-sys-fs\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddbp\" (UniqueName: \"kubernetes.io/projected/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-kube-api-access-cddbp\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410243 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-kubelet\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-cni-bin\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410284 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-node-log\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410318 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cnibin\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c921956-e37e-4cb1-920c-bd7736f44fa1-konnectivity-ca\") pod \"konnectivity-agent-m2hqd\" (UID: \"2c921956-e37e-4cb1-920c-bd7736f44fa1\") " pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-system-cni-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-lib-modules\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-var-lib-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-device-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410515 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-etc-kubernetes\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410567 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzwl\" (UniqueName: \"kubernetes.io/projected/071211d6-06de-48c3-8f5e-fd7b5edc77d7-kube-api-access-6pzwl\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.411203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysconfig\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410604 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-tmp-dir\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-systemd\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-run-ovn-kubernetes\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7940f71f-a825-4461-abb6-4f7f87919491-ovn-node-metrics-cert\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410695 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/156ff3fc-7c3f-49d5-a362-949d57777cf1-serviceca\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxpp4\" (UniqueName: \"kubernetes.io/projected/7940f71f-a825-4461-abb6-4f7f87919491-kube-api-access-wxpp4\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-cni-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410797 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-netns\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-multus-certs\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410845 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-cni-binary-copy\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-sys\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-ovn\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410894 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-run\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.410933 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:44.412026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-host\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dea6475c-946b-489d-8730-8a4781dfe17c-etc-tuned\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411008 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-cni-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411013 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbb89330-d651-44e6-b2cf-2f813c42aa6b-host-slash\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.411044 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:44.911022853 +0000 UTC m=+2.100796234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411054 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbb89330-d651-44e6-b2cf-2f813c42aa6b-host-slash\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411060 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-netns\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-systemd-units\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.410849 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-cni-bin\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-sys\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-slash\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-run-multus-certs\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-registration-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-env-overrides\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411137 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/156ff3fc-7c3f-49d5-a362-949d57777cf1-host\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411159 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-etc-selinux\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-kubernetes\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.412938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-host\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411205 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972t2\" (UniqueName: \"kubernetes.io/projected/71902b90-5bb9-481e-9c6a-08cef1630d47-kube-api-access-972t2\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-hostroot\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411257 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-modprobe-d\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411039 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysctl-d\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d1534dc-38b2-4881-9fd1-80795b1e4b24-dbus\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411343 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-systemd-units\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-systemd\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411384 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-systemd\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-hostroot\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411513 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-env-overrides\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411523 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-modprobe-d\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411566 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-run-netns\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411600 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysctl-d\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411603 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-ovn\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411134 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-slash\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.413759 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411651 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-os-release\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411666 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d1534dc-38b2-4881-9fd1-80795b1e4b24-dbus\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411690 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-cni-bin\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411698 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c921956-e37e-4cb1-920c-bd7736f44fa1-konnectivity-ca\") pod \"konnectivity-agent-m2hqd\" (UID: \"2c921956-e37e-4cb1-920c-bd7736f44fa1\") " pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411717 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-kubernetes\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411722 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-etc-selinux\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411778 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-var-lib-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411782 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-run\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-run-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411849 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-run-ovn-kubernetes\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.411860 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71902b90-5bb9-481e-9c6a-08cef1630d47-device-dir\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.411932 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:44.911914687 +0000 UTC m=+2.101688052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411954 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysconfig\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.411977 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-etc-kubernetes\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412056 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-systemd\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.414510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412140 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-lib-modules\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412240 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-log-socket\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-ovnkube-config\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412358 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-cni-multus\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-kubelet\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-conf-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412450 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-system-cni-dir\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/071211d6-06de-48c3-8f5e-fd7b5edc77d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412493 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-cni-multus\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412501 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-system-cni-dir\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysctl-conf\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412546 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-log-socket\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412557 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-cnibin\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412551 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-multus-conf-dir\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.414981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412583 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-os-release\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-host-var-lib-kubelet\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-var-lib-kubelet\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbfd\" (UniqueName: \"kubernetes.io/projected/dea6475c-946b-489d-8730-8a4781dfe17c-kube-api-access-qdbfd\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412666 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-etc-sysctl-conf\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-hosts-file\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-etc-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412717 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/071211d6-06de-48c3-8f5e-fd7b5edc77d7-os-release\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412760 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-cnibin\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412764 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7940f71f-a825-4461-abb6-4f7f87919491-etc-openvswitch\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dea6475c-946b-489d-8730-8a4781dfe17c-var-lib-kubelet\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.412843 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-hosts-file\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.413092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7940f71f-a825-4461-abb6-4f7f87919491-ovnkube-config\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.413598 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea6475c-946b-489d-8730-8a4781dfe17c-tmp\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.413662 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dea6475c-946b-489d-8730-8a4781dfe17c-etc-tuned\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.413767 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c921956-e37e-4cb1-920c-bd7736f44fa1-agent-certs\") pod \"konnectivity-agent-m2hqd\" (UID: \"2c921956-e37e-4cb1-920c-bd7736f44fa1\") " pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.415509 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.413990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7940f71f-a825-4461-abb6-4f7f87919491-ovn-node-metrics-cert\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.419100 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.419078 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:44.419207 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.419107 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:44.419207 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.419120 2572 projected.go:196] Error preparing data for projected volume kube-api-access-lnkw6 for pod openshift-network-diagnostics/network-check-target-k6847: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:44.419207 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.419183 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6 podName:d3d690c0-0665-4922-9786-e2ab30ea50b9 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:44.919167494 +0000 UTC m=+2.108940874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lnkw6" (UniqueName: "kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6") pod "network-check-target-k6847" (UID: "d3d690c0-0665-4922-9786-e2ab30ea50b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:44.420425 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.420011 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddbp\" (UniqueName: \"kubernetes.io/projected/aca3ad49-96eb-4f52-b11e-4bea99d26f5d-kube-api-access-cddbp\") pod \"node-resolver-npp2t\" (UID: \"aca3ad49-96eb-4f52-b11e-4bea99d26f5d\") " pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.420425 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.420364 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-972t2\" (UniqueName: \"kubernetes.io/projected/71902b90-5bb9-481e-9c6a-08cef1630d47-kube-api-access-972t2\") pod \"aws-ebs-csi-driver-node-rpdfr\" (UID: \"71902b90-5bb9-481e-9c6a-08cef1630d47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.420574 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.420492 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtlt\" (UniqueName: \"kubernetes.io/projected/bbb89330-d651-44e6-b2cf-2f813c42aa6b-kube-api-access-6jtlt\") pod \"iptables-alerter-thtn7\" (UID: \"bbb89330-d651-44e6-b2cf-2f813c42aa6b\") " pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.421100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.421082 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrqr\" (UniqueName: \"kubernetes.io/projected/156ff3fc-7c3f-49d5-a362-949d57777cf1-kube-api-access-svrqr\") pod \"node-ca-tdhg2\" (UID: \"156ff3fc-7c3f-49d5-a362-949d57777cf1\") " pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.421346 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.421321 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxpp4\" (UniqueName: \"kubernetes.io/projected/7940f71f-a825-4461-abb6-4f7f87919491-kube-api-access-wxpp4\") pod \"ovnkube-node-rpnss\" (UID: \"7940f71f-a825-4461-abb6-4f7f87919491\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.421570 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.421548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqlg\" (UniqueName: \"kubernetes.io/projected/fe4eaf13-1705-4a64-9c91-efaa0ceaff09-kube-api-access-xnqlg\") pod \"multus-wm8lw\" (UID: \"fe4eaf13-1705-4a64-9c91-efaa0ceaff09\") " pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.421853 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.421836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbfd\" (UniqueName: \"kubernetes.io/projected/dea6475c-946b-489d-8730-8a4781dfe17c-kube-api-access-qdbfd\") pod \"tuned-ckr54\" (UID: \"dea6475c-946b-489d-8730-8a4781dfe17c\") " pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.422272 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.422251 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzwl\" (UniqueName: \"kubernetes.io/projected/071211d6-06de-48c3-8f5e-fd7b5edc77d7-kube-api-access-6pzwl\") pod \"multus-additional-cni-plugins-gljn7\" (UID: \"071211d6-06de-48c3-8f5e-fd7b5edc77d7\") " pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.422580 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.422563 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmlv\" (UniqueName: \"kubernetes.io/projected/61deeab5-0b0e-4a34-80ea-e476ec574ee7-kube-api-access-5lmlv\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.453524 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.453499 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76db135eb9a95b201fdaec547a8e38e2.slice/crio-3d83a02d908405e017cfcb3182c3e73850374db93656d9fb68c1b9108ea542bc WatchSource:0}: Error finding container 3d83a02d908405e017cfcb3182c3e73850374db93656d9fb68c1b9108ea542bc: Status 404 returned error can't find the container with id 3d83a02d908405e017cfcb3182c3e73850374db93656d9fb68c1b9108ea542bc Jul 28 17:08:44.453782 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.453765 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28b7e1bb7bab65a1df8ef56c8ebc188.slice/crio-43096838ca9825f7222b1f658a5a0c3c2b4cb84728ac571a87311cd60ffa4563 WatchSource:0}: Error finding container 43096838ca9825f7222b1f658a5a0c3c2b4cb84728ac571a87311cd60ffa4563: Status 404 returned error can't find the container with id 43096838ca9825f7222b1f658a5a0c3c2b4cb84728ac571a87311cd60ffa4563 Jul 28 17:08:44.459409 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.459381 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:08:44.525304 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.525244 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:08:44.531367 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.531348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" Jul 28 17:08:44.535023 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.535000 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c921956_e37e_4cb1_920c_bd7736f44fa1.slice/crio-bf55a35af348e17c631b1d37411faf2b327b5f6c4dd9341838e0af1ed1f6d220 WatchSource:0}: Error finding container bf55a35af348e17c631b1d37411faf2b327b5f6c4dd9341838e0af1ed1f6d220: Status 404 returned error can't find the container with id bf55a35af348e17c631b1d37411faf2b327b5f6c4dd9341838e0af1ed1f6d220 Jul 28 17:08:44.540053 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.540035 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71902b90_5bb9_481e_9c6a_08cef1630d47.slice/crio-1e0685c39544bbf5f02ab31726d096b23bdfe4908f68c581d0f02df9a77b63d0 WatchSource:0}: Error finding container 1e0685c39544bbf5f02ab31726d096b23bdfe4908f68c581d0f02df9a77b63d0: Status 404 returned error can't find the container with id 1e0685c39544bbf5f02ab31726d096b23bdfe4908f68c581d0f02df9a77b63d0 Jul 28 17:08:44.563590 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.563572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ckr54" Jul 28 17:08:44.566688 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.566672 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npp2t" Jul 28 17:08:44.573860 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.573825 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea6475c_946b_489d_8730_8a4781dfe17c.slice/crio-ec01708dc46b65622d2a0862e4f1223c360a8bb8ac225c4f569774eea67a3060 WatchSource:0}: Error finding container ec01708dc46b65622d2a0862e4f1223c360a8bb8ac225c4f569774eea67a3060: Status 404 returned error can't find the container with id ec01708dc46b65622d2a0862e4f1223c360a8bb8ac225c4f569774eea67a3060 Jul 28 17:08:44.576527 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.576509 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca3ad49_96eb_4f52_b11e_4bea99d26f5d.slice/crio-07062a722cf6a618d386655c5a4f9dc277d6811b7f30b8430bd159c2bc1f44a8 WatchSource:0}: Error finding container 07062a722cf6a618d386655c5a4f9dc277d6811b7f30b8430bd159c2bc1f44a8: Status 404 returned error can't find the container with id 07062a722cf6a618d386655c5a4f9dc277d6811b7f30b8430bd159c2bc1f44a8 Jul 28 17:08:44.579569 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.579555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wm8lw" Jul 28 17:08:44.584180 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.584165 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thtn7" Jul 28 17:08:44.587758 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.587741 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4eaf13_1705_4a64_9c91_efaa0ceaff09.slice/crio-b0b5bbba6c57a24e6f368071454dc1ff990641fd94d5b4f0d0d29ec7e6ea4fbe WatchSource:0}: Error finding container b0b5bbba6c57a24e6f368071454dc1ff990641fd94d5b4f0d0d29ec7e6ea4fbe: Status 404 returned error can't find the container with id b0b5bbba6c57a24e6f368071454dc1ff990641fd94d5b4f0d0d29ec7e6ea4fbe Jul 28 17:08:44.592249 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.592232 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb89330_d651_44e6_b2cf_2f813c42aa6b.slice/crio-68f7378880b0a7f9cd38957c83ed913998942744dbb37a0eb6514f1d3429388f WatchSource:0}: Error finding container 68f7378880b0a7f9cd38957c83ed913998942744dbb37a0eb6514f1d3429388f: Status 404 returned error can't find the container with id 68f7378880b0a7f9cd38957c83ed913998942744dbb37a0eb6514f1d3429388f Jul 28 17:08:44.602239 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.602225 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:44.603563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.603548 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:08:44.612612 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.612595 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7940f71f_a825_4461_abb6_4f7f87919491.slice/crio-9cdb410f7223308b68d93176b44c96cc17f74a0ef519d760644ca072120db3f0 WatchSource:0}: Error finding container 9cdb410f7223308b68d93176b44c96cc17f74a0ef519d760644ca072120db3f0: Status 404 returned error can't find the container with id 9cdb410f7223308b68d93176b44c96cc17f74a0ef519d760644ca072120db3f0 Jul 28 17:08:44.620786 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.620768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tdhg2" Jul 28 17:08:44.625474 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.625453 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gljn7" Jul 28 17:08:44.628306 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.628288 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156ff3fc_7c3f_49d5_a362_949d57777cf1.slice/crio-8ccb9c897414f31c6c6abd24c55bf0774aa810d95588f18dff701be30516f059 WatchSource:0}: Error finding container 8ccb9c897414f31c6c6abd24c55bf0774aa810d95588f18dff701be30516f059: Status 404 returned error can't find the container with id 8ccb9c897414f31c6c6abd24c55bf0774aa810d95588f18dff701be30516f059 Jul 28 17:08:44.633557 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:08:44.633538 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071211d6_06de_48c3_8f5e_fd7b5edc77d7.slice/crio-e4be18c05c6c72224113b3f8a3fd8d1be266e089f676e53a21fc153e6f81f9ae WatchSource:0}: Error finding container e4be18c05c6c72224113b3f8a3fd8d1be266e089f676e53a21fc153e6f81f9ae: Status 404 returned error can't find the container with id e4be18c05c6c72224113b3f8a3fd8d1be266e089f676e53a21fc153e6f81f9ae Jul 28 17:08:44.635009 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.634990 2572 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-gljn7_openshift-multus(071211d6-06de-48c3-8f5e-fd7b5edc77d7): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 28 17:08:44.635078 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.635024 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-gljn7" podUID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" Jul 28 17:08:44.916279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.916191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:44.916279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:44.916234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:44.916509 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.916329 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:44.916509 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.916404 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:45.91637042 +0000 UTC m=+3.106143785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:44.916509 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.916330 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:44.916509 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:44.916464 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:45.91645049 +0000 UTC m=+3.106223853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:45.017757 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.017163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:45.017757 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.017319 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:45.017757 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.017338 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:45.017757 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.017349 2572 projected.go:196] Error preparing data for projected volume kube-api-access-lnkw6 for pod openshift-network-diagnostics/network-check-target-k6847: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:45.017757 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.017427 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6 podName:d3d690c0-0665-4922-9786-e2ab30ea50b9 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:46.017407708 +0000 UTC m=+3.207181093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnkw6" (UniqueName: "kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6") pod "network-check-target-k6847" (UID: "d3d690c0-0665-4922-9786-e2ab30ea50b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:45.349196 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.349102 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:03:44 +0000 UTC" deadline="2028-02-10 01:36:41.06819997 +0000 UTC" Jul 28 17:08:45.349196 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.349146 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13472h27m55.719065327s" Jul 28 17:08:45.363573 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.363517 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npp2t" event={"ID":"aca3ad49-96eb-4f52-b11e-4bea99d26f5d","Type":"ContainerStarted","Data":"07062a722cf6a618d386655c5a4f9dc277d6811b7f30b8430bd159c2bc1f44a8"} Jul 28 17:08:45.385679 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.385645 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ckr54" event={"ID":"dea6475c-946b-489d-8730-8a4781dfe17c","Type":"ContainerStarted","Data":"ec01708dc46b65622d2a0862e4f1223c360a8bb8ac225c4f569774eea67a3060"} Jul 28 17:08:45.393798 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.393761 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tdhg2" event={"ID":"156ff3fc-7c3f-49d5-a362-949d57777cf1","Type":"ContainerStarted","Data":"8ccb9c897414f31c6c6abd24c55bf0774aa810d95588f18dff701be30516f059"} Jul 28 17:08:45.409883 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.409852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wm8lw" event={"ID":"fe4eaf13-1705-4a64-9c91-efaa0ceaff09","Type":"ContainerStarted","Data":"b0b5bbba6c57a24e6f368071454dc1ff990641fd94d5b4f0d0d29ec7e6ea4fbe"} Jul 28 17:08:45.423226 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.423196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" event={"ID":"71902b90-5bb9-481e-9c6a-08cef1630d47","Type":"ContainerStarted","Data":"1e0685c39544bbf5f02ab31726d096b23bdfe4908f68c581d0f02df9a77b63d0"} Jul 28 17:08:45.425567 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.425497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m2hqd" event={"ID":"2c921956-e37e-4cb1-920c-bd7736f44fa1","Type":"ContainerStarted","Data":"bf55a35af348e17c631b1d37411faf2b327b5f6c4dd9341838e0af1ed1f6d220"} Jul 28 17:08:45.430265 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.430239 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" event={"ID":"76db135eb9a95b201fdaec547a8e38e2","Type":"ContainerStarted","Data":"3d83a02d908405e017cfcb3182c3e73850374db93656d9fb68c1b9108ea542bc"} Jul 28 17:08:45.449525 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.449467 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" event={"ID":"a28b7e1bb7bab65a1df8ef56c8ebc188","Type":"ContainerStarted","Data":"43096838ca9825f7222b1f658a5a0c3c2b4cb84728ac571a87311cd60ffa4563"} Jul 28 17:08:45.460416 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.460380 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerStarted","Data":"e4be18c05c6c72224113b3f8a3fd8d1be266e089f676e53a21fc153e6f81f9ae"} Jul 28 17:08:45.462750 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.462722 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-gljn7" podUID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" Jul 28 17:08:45.463598 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.463573 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"9cdb410f7223308b68d93176b44c96cc17f74a0ef519d760644ca072120db3f0"} Jul 28 17:08:45.489052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.488980 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thtn7" event={"ID":"bbb89330-d651-44e6-b2cf-2f813c42aa6b","Type":"ContainerStarted","Data":"68f7378880b0a7f9cd38957c83ed913998942744dbb37a0eb6514f1d3429388f"} Jul 28 17:08:45.672558 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.672479 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:45.924483 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.924376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:45.924483 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:45.924478 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:45.924693 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.924592 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:45.924693 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.924630 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:45.924693 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.924651 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:47.924632665 +0000 UTC m=+5.114406031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:45.924693 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:45.924672 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:47.924657436 +0000 UTC m=+5.114430814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:46.025119 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.025086 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:46.025324 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.025299 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:46.025324 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.025320 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:46.025469 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.025334 2572 projected.go:196] Error preparing data for projected volume kube-api-access-lnkw6 for pod openshift-network-diagnostics/network-check-target-k6847: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:46.025469 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.025402 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6 podName:d3d690c0-0665-4922-9786-e2ab30ea50b9 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:48.025370631 +0000 UTC m=+5.215144008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnkw6" (UniqueName: "kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6") pod "network-check-target-k6847" (UID: "d3d690c0-0665-4922-9786-e2ab30ea50b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:46.331608 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.331574 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:46.331795 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.331708 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:46.332110 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.332091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:46.332219 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.332185 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:46.332274 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.332255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:46.332423 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:46.332357 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:46.349876 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.349665 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:03:44 +0000 UTC" deadline="2028-03-12 00:25:26.526414099 +0000 UTC" Jul 28 17:08:46.349876 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.349692 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14215h16m40.17672575s" Jul 28 17:08:46.593824 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:46.593745 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:47.139492 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:47.139462 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:47.949215 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:47.945572 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:47.949215 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:47.945635 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:47.949215 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:47.946252 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:47.949215 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:47.946338 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:51.946317747 +0000 UTC m=+9.136091113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:47.949215 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:47.946929 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:47.949215 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:47.947059 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:51.946995344 +0000 UTC m=+9.136768721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:48.046858 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:48.046822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:48.047042 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.047005 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:48.047042 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.047023 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:48.047042 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.047037 2572 projected.go:196] Error preparing data for projected volume kube-api-access-lnkw6 for pod openshift-network-diagnostics/network-check-target-k6847: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:48.047202 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.047089 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6 podName:d3d690c0-0665-4922-9786-e2ab30ea50b9 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:52.047072087 +0000 UTC m=+9.236845453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnkw6" (UniqueName: "kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6") pod "network-check-target-k6847" (UID: "d3d690c0-0665-4922-9786-e2ab30ea50b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:48.332403 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:48.332361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:48.332645 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:48.332408 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:48.332645 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.332509 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:48.332645 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.332558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:48.332645 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:48.332626 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:48.332897 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:48.332707 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:50.332001 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:50.331973 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:50.332624 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:50.332007 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:50.332624 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:50.332114 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:50.332624 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:50.332195 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:50.332624 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:50.332244 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:50.332624 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:50.332311 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:51.978607 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:51.978573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:51.978983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:51.978619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:51.978983 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:51.978735 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:51.978983 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:51.978750 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:51.978983 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:51.978805 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:59.978784098 +0000 UTC m=+17.168557478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:51.978983 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:51.978823 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:59.97881404 +0000 UTC m=+17.168587409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:52.079418 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:52.079150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:52.079418 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.079371 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:52.079418 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.079412 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:52.079418 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.079424 2572 projected.go:196] Error preparing data for projected volume kube-api-access-lnkw6 for pod openshift-network-diagnostics/network-check-target-k6847: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:52.079734 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.079477 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6 podName:d3d690c0-0665-4922-9786-e2ab30ea50b9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:00.079460593 +0000 UTC m=+17.269233973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnkw6" (UniqueName: "kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6") pod "network-check-target-k6847" (UID: "d3d690c0-0665-4922-9786-e2ab30ea50b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:52.332066 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:52.331715 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:52.332066 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:52.331732 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:52.332066 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.331868 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:52.332066 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.331930 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:52.332066 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:52.331988 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:52.332066 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:52.332067 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:54.331853 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:54.331821 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:54.332309 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:54.331825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:54.332309 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:54.331939 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:54.332309 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:54.331949 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:54.332309 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:54.332037 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:54.332309 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:54.332122 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:56.331633 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:56.331599 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:56.331633 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:56.331631 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:56.332280 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:56.331599 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:56.332280 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:56.331719 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:56.332280 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:56.331866 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:56.332280 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:56.331946 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:08:58.332371 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:58.332332 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:08:58.332852 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:58.332337 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:08:58.332852 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:58.332477 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:08:58.332852 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:08:58.332341 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:08:58.332852 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:58.332556 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:08:58.332852 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:08:58.332646 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:00.039475 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:00.039443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:00.039475 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:00.039480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:00.039940 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.039600 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:00.039940 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.039657 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:16.0396399 +0000 UTC m=+33.229413275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:00.039940 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.039600 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:00.039940 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.039743 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:16.039719486 +0000 UTC m=+33.229492880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:00.140038 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:00.140006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:00.140194 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.140138 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:00.140194 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.140156 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:00.140194 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.140164 2572 projected.go:196] Error preparing data for projected volume kube-api-access-lnkw6 for pod openshift-network-diagnostics/network-check-target-k6847: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:00.140309 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.140209 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6 podName:d3d690c0-0665-4922-9786-e2ab30ea50b9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:16.140197077 +0000 UTC m=+33.329970454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnkw6" (UniqueName: "kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6") pod "network-check-target-k6847" (UID: "d3d690c0-0665-4922-9786-e2ab30ea50b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:00.332407 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:00.332327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:00.332407 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:00.332363 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:00.332407 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:00.332327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:00.332664 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.332459 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:00.332664 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.332599 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:00.332797 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:00.332769 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:02.332420 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:02.332370 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:02.332967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:02.332373 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:02.332967 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:02.332616 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:02.332967 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:02.332502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:02.332967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:02.332385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:02.332967 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:02.332727 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:03.539699 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.539430 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wm8lw" event={"ID":"fe4eaf13-1705-4a64-9c91-efaa0ceaff09","Type":"ContainerStarted","Data":"b77c998dac0b2c3f8d3fd869a1593b57b21e477efe50bd99700fde6c1b74d48d"} Jul 28 17:09:03.549580 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.549533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" event={"ID":"a28b7e1bb7bab65a1df8ef56c8ebc188","Type":"ContainerStarted","Data":"3fd9d213e8de976af6916aa7fd46ff647add41d18a5115d2b0827cbd99dceb80"} Jul 28 17:09:03.555914 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.555770 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"7ad276b3ccf598573afe852179963369e73794f17c063596887febdccec267b9"} Jul 28 17:09:03.555914 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.555858 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"08129d60dd75df81b0fceb6730b8efc0bb490b69c5ed70846e5e72760d7e998b"} Jul 28 17:09:03.561658 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.561605 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ckr54" event={"ID":"dea6475c-946b-489d-8730-8a4781dfe17c","Type":"ContainerStarted","Data":"aeb3798d8db8ae6262f9ae68ab2a952b1931b75ec3b010da99d786968699fc6c"} Jul 28 17:09:03.562041 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.561813 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wm8lw" podStartSLOduration=1.937292021 podStartE2EDuration="20.56137475s" totalImagesPullingTime="18.624082729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:03.56015272 +0000 UTC m=+20.749926106" watchObservedRunningTime="2026-07-28 17:09:03.56137475 +0000 UTC m=+20.751148135" Jul 28 17:09:03.577216 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.576355 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-138.ec2.internal" podStartSLOduration=19.576339776 podStartE2EDuration="19.576339776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:03.57617845 +0000 UTC m=+20.765951900" watchObservedRunningTime="2026-07-28 17:09:03.576339776 +0000 UTC m=+20.766113162" Jul 28 17:09:03.592748 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:03.592692 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ckr54" podStartSLOduration=1.9933204199999999 podStartE2EDuration="20.592671039s" totalImagesPullingTime="18.599350619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:03.592572545 +0000 UTC m=+20.782345930" watchObservedRunningTime="2026-07-28 17:09:03.592671039 +0000 UTC m=+20.782444429" Jul 28 17:09:04.331600 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.331565 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:04.331600 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.331601 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:04.331784 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.331680 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:04.331784 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:04.331701 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:04.331784 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:04.331748 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:04.331912 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:04.331837 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:04.567914 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.567882 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"425cb08784e8ad51ef50ae8ee97a86ab975e7a900725f79652b737f6d474e28a"} Jul 28 17:09:04.568714 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.567956 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"d382196cd90b94c274f70087e8255ea37c12962df295d93a1f651a9f9e864786"} Jul 28 17:09:04.568714 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.568003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"2e9155e01ea8a1c326aa36ae11497501413ef6d8d316fabe4741e2ab13b6764e"} Jul 28 17:09:04.568714 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.568038 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"256d6364ad506d71be84a03ceb77d37b69d41218d5a498e0a86ede3016fcd26a"} Jul 28 17:09:04.569855 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.569829 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npp2t" event={"ID":"aca3ad49-96eb-4f52-b11e-4bea99d26f5d","Type":"ContainerStarted","Data":"c64ebae955365515616a1ef933815e2e9940e00bbea5e220c8cf8801a0bdc51a"} Jul 28 17:09:04.571404 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.571366 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tdhg2" event={"ID":"156ff3fc-7c3f-49d5-a362-949d57777cf1","Type":"ContainerStarted","Data":"11b7011016f41547d56b42d205cf8d742a1a4ed437cf9b51e69088a8e75ed093"} Jul 28 17:09:04.573160 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.573129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" event={"ID":"71902b90-5bb9-481e-9c6a-08cef1630d47","Type":"ContainerStarted","Data":"709afed9bba9df5253a9ae48a58cdc746cc202b0b3a198a57d79814935ec5ff2"} Jul 28 17:09:04.574617 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.574596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m2hqd" event={"ID":"2c921956-e37e-4cb1-920c-bd7736f44fa1","Type":"ContainerStarted","Data":"904010ff8d8f4c41369fcf907e57f4241a9092f2f5b31f86843367f9e3fbafdf"} Jul 28 17:09:04.576292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.576273 2572 generic.go:356] "Generic (PLEG): container finished" podID="76db135eb9a95b201fdaec547a8e38e2" containerID="2f63e0e3fd57e5bf73b4486a7f4aae5f98ad62fab01ebaf6e886078ebe5fc327" exitCode=0 Jul 28 17:09:04.576407 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.576376 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" event={"ID":"76db135eb9a95b201fdaec547a8e38e2","Type":"ContainerDied","Data":"2f63e0e3fd57e5bf73b4486a7f4aae5f98ad62fab01ebaf6e886078ebe5fc327"} Jul 28 17:09:04.599709 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.599664 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m2hqd" podStartSLOduration=11.167801375 podStartE2EDuration="21.599650367s" totalImagesPullingTime="10.431848992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:04.599504454 +0000 UTC m=+21.789277838" watchObservedRunningTime="2026-07-28 17:09:04.599650367 +0000 UTC m=+21.789423749" Jul 28 17:09:04.599883 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.599862 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-npp2t" podStartSLOduration=3.017887505 podStartE2EDuration="21.599854979s" totalImagesPullingTime="18.581967474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:04.584688565 +0000 UTC m=+21.774461955" watchObservedRunningTime="2026-07-28 17:09:04.599854979 +0000 UTC m=+21.789628408" Jul 28 17:09:04.619008 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:04.618957 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tdhg2" podStartSLOduration=3.088288896 podStartE2EDuration="21.618939273s" totalImagesPullingTime="18.530650377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:04.618556969 +0000 UTC m=+21.808330354" watchObservedRunningTime="2026-07-28 17:09:04.618939273 +0000 UTC m=+21.808712711" Jul 28 17:09:05.409595 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.409371 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 17:09:05.579829 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.579798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" event={"ID":"71902b90-5bb9-481e-9c6a-08cef1630d47","Type":"ContainerStarted","Data":"8072a0cce3dd167f829cc30c40d9b53d1a0680c9b0157444bf6d0536b2c3df88"} Jul 28 17:09:05.581437 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.581414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" event={"ID":"76db135eb9a95b201fdaec547a8e38e2","Type":"ContainerStarted","Data":"91fd893985f49cea143a450ed4793d0e9b2257d6008de5a28f54985a55685737"} Jul 28 17:09:05.582852 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.582830 2572 generic.go:356] "Generic (PLEG): container finished" podID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" containerID="46dfa5813b39caecaf65a37f74041a026fc89c992541803dc9f6a2fa94b16d09" exitCode=0 Jul 28 17:09:05.582941 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.582902 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerDied","Data":"46dfa5813b39caecaf65a37f74041a026fc89c992541803dc9f6a2fa94b16d09"} Jul 28 17:09:05.584238 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.584192 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thtn7" event={"ID":"bbb89330-d651-44e6-b2cf-2f813c42aa6b","Type":"ContainerStarted","Data":"346cc7795d0860d3d5e49c00f8c8779f64a2c290386bcac8f121c78f899ca279"} Jul 28 17:09:05.610544 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.610505 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-138.ec2.internal" podStartSLOduration=21.610493741 podStartE2EDuration="21.610493741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:05.597058076 +0000 UTC m=+22.786831461" watchObservedRunningTime="2026-07-28 17:09:05.610493741 +0000 UTC m=+22.800267125" Jul 28 17:09:05.630344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:05.630309 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-thtn7" podStartSLOduration=4.050689225 podStartE2EDuration="22.630297904s" totalImagesPullingTime="18.579608679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:05.610304798 +0000 UTC m=+22.800078184" watchObservedRunningTime="2026-07-28 17:09:05.630297904 +0000 UTC m=+22.820071289" Jul 28 17:09:06.262331 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.262179 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:09:05.409593661Z","UUID":"158394f5-3200-41c9-a92d-7990eb17289b","Handler":null,"Name":"","Endpoint":""} Jul 28 17:09:06.263730 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.263693 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 17:09:06.263909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.263782 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 17:09:06.332307 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.332279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:06.332470 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.332320 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:06.332470 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.332292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:06.332470 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:06.332426 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:06.332635 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:06.332513 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:06.332635 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:06.332605 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:06.589876 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.589789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"8e75853a7ebb24b4b7070d95925fbde03819e5aab94b68e29c458974cb4ef704"} Jul 28 17:09:06.624970 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.624943 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:09:06.626020 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:06.626000 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:09:07.594805 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:07.594689 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" event={"ID":"71902b90-5bb9-481e-9c6a-08cef1630d47","Type":"ContainerStarted","Data":"457b3ee71bb7a84d38e53bf94b15ec3529e163a5035f1626483363bc344c29f3"} Jul 28 17:09:07.594805 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:07.594762 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:09:07.595362 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:07.595081 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m2hqd" Jul 28 17:09:07.624828 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:07.624783 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rpdfr" podStartSLOduration=3.023101575 podStartE2EDuration="24.624769516s" totalImagesPullingTime="21.601667941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:07.624097719 +0000 UTC m=+24.813871106" watchObservedRunningTime="2026-07-28 17:09:07.624769516 +0000 UTC m=+24.814542901" Jul 28 17:09:08.332089 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:08.331909 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:08.332250 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:08.331952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:08.332250 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:08.332169 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:08.332250 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:08.331979 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:08.332250 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:08.332242 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:08.332477 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:08.332342 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:10.332231 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.332205 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:10.332830 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.332299 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:10.332830 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:10.332307 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:10.332830 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.332324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:10.332830 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:10.332406 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:10.332830 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:10.332462 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:10.602464 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.602436 2572 generic.go:356] "Generic (PLEG): container finished" podID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" containerID="3675f6f8cb4e2a5628803f92770c4a1ac89516d64c89bac80b4ef092efcf2dcc" exitCode=0 Jul 28 17:09:10.602601 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.602493 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerDied","Data":"3675f6f8cb4e2a5628803f92770c4a1ac89516d64c89bac80b4ef092efcf2dcc"} Jul 28 17:09:10.605998 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.605977 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" event={"ID":"7940f71f-a825-4461-abb6-4f7f87919491","Type":"ContainerStarted","Data":"aa1ed5ed72fe3dfba65b14240de0c6968837c163abdd9a972cac7cb520d2be7f"} Jul 28 17:09:10.606237 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.606213 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:09:10.606334 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.606279 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:09:10.606334 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.606327 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:09:10.620929 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.620907 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:09:10.621039 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.621028 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:09:10.648398 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:10.648342 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" podStartSLOduration=8.957435339 podStartE2EDuration="27.648329949s" totalImagesPullingTime="18.69089461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:10.647344689 +0000 UTC m=+27.837118072" watchObservedRunningTime="2026-07-28 17:09:10.648329949 +0000 UTC m=+27.838103334" Jul 28 17:09:12.098143 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.097062 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nc9d4"] Jul 28 17:09:12.098143 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.097465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:12.098143 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:12.097593 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:12.099801 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.099777 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k6847"] Jul 28 17:09:12.099900 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.099861 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:12.099952 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:12.099924 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:12.116688 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.116668 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4kwh"] Jul 28 17:09:12.116790 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.116772 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:12.116890 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:12.116871 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:12.612982 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.612951 2572 generic.go:356] "Generic (PLEG): container finished" podID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" containerID="8b16b2cc780450097e89be19a3e12b7c7549868ad10ed52de5d4c9c3c639fe88" exitCode=0 Jul 28 17:09:12.613157 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:12.613023 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerDied","Data":"8b16b2cc780450097e89be19a3e12b7c7549868ad10ed52de5d4c9c3c639fe88"} Jul 28 17:09:13.333102 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:13.333067 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:13.333639 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:13.333191 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:14.331401 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:14.331360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:14.331401 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:14.331380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:14.331591 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:14.331475 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:09:14.331591 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:14.331577 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nc9d4" podUID="0d1534dc-38b2-4881-9fd1-80795b1e4b24" Jul 28 17:09:14.622825 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:14.622751 2572 generic.go:356] "Generic (PLEG): container finished" podID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" containerID="7f42c3034b2a933453a5221eb7724748562f67571cf0583d40769d96a5d9a2a6" exitCode=0 Jul 28 17:09:14.622825 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:14.622798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerDied","Data":"7f42c3034b2a933453a5221eb7724748562f67571cf0583d40769d96a5d9a2a6"} Jul 28 17:09:15.332102 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:15.332071 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:15.332355 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:15.332188 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k6847" podUID="d3d690c0-0665-4922-9786-e2ab30ea50b9" Jul 28 17:09:16.063556 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.063253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:16.063556 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.063544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:16.064023 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.063415 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:16.064023 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.063642 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:48.063627545 +0000 UTC m=+65.253400912 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:16.064023 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.063675 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:16.064023 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.063731 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret podName:0d1534dc-38b2-4881-9fd1-80795b1e4b24 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:48.063714477 +0000 UTC m=+65.253487845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret") pod "global-pull-secret-syncer-nc9d4" (UID: "0d1534dc-38b2-4881-9fd1-80795b1e4b24") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:16.114538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.114512 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-138.ec2.internal" event="NodeReady" Jul 28 17:09:16.114673 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.114650 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 17:09:16.153097 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.153073 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65d68454b6-wdf5b"] Jul 28 17:09:16.154867 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.154844 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.155102 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.155080 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p"] Jul 28 17:09:16.156599 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.156583 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" Jul 28 17:09:16.156853 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.156828 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 17:09:16.157019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.157004 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2dfz6\"" Jul 28 17:09:16.157084 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.157047 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 17:09:16.157139 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.157089 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 17:09:16.159159 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.158903 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:09:16.159159 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.158995 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-48mpf\"" Jul 28 17:09:16.159358 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.159330 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:09:16.163438 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.163421 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 17:09:16.163861 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.163839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:16.169377 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.169352 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p"] Jul 28 17:09:16.169503 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.169445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkw6\" (UniqueName: \"kubernetes.io/projected/d3d690c0-0665-4922-9786-e2ab30ea50b9-kube-api-access-lnkw6\") pod \"network-check-target-k6847\" (UID: \"d3d690c0-0665-4922-9786-e2ab30ea50b9\") " pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:16.172613 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.172339 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6kxfc"] Jul 28 17:09:16.174664 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.174507 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r4pld"] Jul 28 17:09:16.174750 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.174738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.177843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.177441 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 17:09:16.177843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.177529 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 17:09:16.177843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.177585 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zj6nk\"" Jul 28 17:09:16.181735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.181715 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d68454b6-wdf5b"] Jul 28 17:09:16.181818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.181808 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.183814 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.183793 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 17:09:16.183931 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.183910 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 17:09:16.183996 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.183798 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6sdzh\"" Jul 28 17:09:16.183996 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.183951 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 17:09:16.184651 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.184632 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4pld"] Jul 28 17:09:16.190335 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.189575 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6kxfc"] Jul 28 17:09:16.264634 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt95s\" (UniqueName: \"kubernetes.io/projected/8ab5e312-2589-4188-b5d1-fe77447c7e52-kube-api-access-dt95s\") pod \"ingress-canary-r4pld\" (UID: \"8ab5e312-2589-4188-b5d1-fe77447c7e52\") " pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.264780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264640 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xk5q\" (UniqueName: \"kubernetes.io/projected/49d74a24-e3fd-4b88-84eb-f2fa8df30516-kube-api-access-2xk5q\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.264780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264661 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.264780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d74a24-e3fd-4b88-84eb-f2fa8df30516-config-volume\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.264780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-certificates\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.264780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-trusted-ca\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-installation-pull-secrets\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-bound-sa-token\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264879 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-ca-trust-extracted\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhnj7\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-kube-api-access-dhnj7\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264974 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-image-registry-private-configuration\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.264992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.265006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d74a24-e3fd-4b88-84eb-f2fa8df30516-tmp-dir\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.265031 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fm5\" (UniqueName: \"kubernetes.io/projected/44b20fb3-a288-49a1-bc95-a87e57017d08-kube-api-access-76fm5\") pod \"network-check-source-7d88cb55d8-wzm7p\" (UID: \"44b20fb3-a288-49a1-bc95-a87e57017d08\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" Jul 28 17:09:16.265076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.265050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ab5e312-2589-4188-b5d1-fe77447c7e52-cert\") pod \"ingress-canary-r4pld\" (UID: \"8ab5e312-2589-4188-b5d1-fe77447c7e52\") " pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.319995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.319922 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m2hqd_2c921956-e37e-4cb1-920c-bd7736f44fa1/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.331705 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.331686 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:16.331893 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.331858 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:16.333913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.333891 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:09:16.334029 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.333998 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-24g9l\"" Jul 28 17:09:16.334096 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.334036 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:09:16.365334 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xk5q\" (UniqueName: \"kubernetes.io/projected/49d74a24-e3fd-4b88-84eb-f2fa8df30516-kube-api-access-2xk5q\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.365507 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.365507 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d74a24-e3fd-4b88-84eb-f2fa8df30516-config-volume\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.365507 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365383 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-certificates\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.365507 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.365494 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:16.365507 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.365510 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:16.365777 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365517 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-trusted-ca\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.365777 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-installation-pull-secrets\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.365777 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.365572 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:09:16.865555056 +0000 UTC m=+34.055328423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:16.365777 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-bound-sa-token\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.365777 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-ca-trust-extracted\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhnj7\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-kube-api-access-dhnj7\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-image-registry-private-configuration\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d74a24-e3fd-4b88-84eb-f2fa8df30516-tmp-dir\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fm5\" (UniqueName: \"kubernetes.io/projected/44b20fb3-a288-49a1-bc95-a87e57017d08-kube-api-access-76fm5\") pod \"network-check-source-7d88cb55d8-wzm7p\" (UID: \"44b20fb3-a288-49a1-bc95-a87e57017d08\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ab5e312-2589-4188-b5d1-fe77447c7e52-cert\") pod \"ingress-canary-r4pld\" (UID: \"8ab5e312-2589-4188-b5d1-fe77447c7e52\") " pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-certificates\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.366032 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt95s\" (UniqueName: \"kubernetes.io/projected/8ab5e312-2589-4188-b5d1-fe77447c7e52-kube-api-access-dt95s\") pod \"ingress-canary-r4pld\" (UID: \"8ab5e312-2589-4188-b5d1-fe77447c7e52\") " pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.366439 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.366239 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-ca-trust-extracted\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.366439 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.365965 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d74a24-e3fd-4b88-84eb-f2fa8df30516-config-volume\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.366545 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.366488 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:16.366545 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.366517 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-trusted-ca\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.366634 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.366547 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:16.866528797 +0000 UTC m=+34.056302164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:16.366812 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.366780 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d74a24-e3fd-4b88-84eb-f2fa8df30516-tmp-dir\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.369028 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.369009 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-installation-pull-secrets\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.369028 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.369017 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-image-registry-private-configuration\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.369160 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.369053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ab5e312-2589-4188-b5d1-fe77447c7e52-cert\") pod \"ingress-canary-r4pld\" (UID: \"8ab5e312-2589-4188-b5d1-fe77447c7e52\") " pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.376019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.375971 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xk5q\" (UniqueName: \"kubernetes.io/projected/49d74a24-e3fd-4b88-84eb-f2fa8df30516-kube-api-access-2xk5q\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.376924 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.376902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fm5\" (UniqueName: \"kubernetes.io/projected/44b20fb3-a288-49a1-bc95-a87e57017d08-kube-api-access-76fm5\") pod \"network-check-source-7d88cb55d8-wzm7p\" (UID: \"44b20fb3-a288-49a1-bc95-a87e57017d08\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" Jul 28 17:09:16.377613 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.377591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt95s\" (UniqueName: \"kubernetes.io/projected/8ab5e312-2589-4188-b5d1-fe77447c7e52-kube-api-access-dt95s\") pod \"ingress-canary-r4pld\" (UID: \"8ab5e312-2589-4188-b5d1-fe77447c7e52\") " pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.377703 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.377619 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhnj7\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-kube-api-access-dhnj7\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.378025 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.378005 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-bound-sa-token\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.406871 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.406835 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m2hqd_2c921956-e37e-4cb1-920c-bd7736f44fa1/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.481964 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.481941 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" Jul 28 17:09:16.497565 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.497532 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4pld" Jul 28 17:09:16.631767 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.631578 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p"] Jul 28 17:09:16.634248 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.634226 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4pld"] Jul 28 17:09:16.635736 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:09:16.635612 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b20fb3_a288_49a1_bc95_a87e57017d08.slice/crio-9c7179705bf9090fc3e6516bbe99c12122af69574892089a9c63de290ba7b3a6 WatchSource:0}: Error finding container 9c7179705bf9090fc3e6516bbe99c12122af69574892089a9c63de290ba7b3a6: Status 404 returned error can't find the container with id 9c7179705bf9090fc3e6516bbe99c12122af69574892089a9c63de290ba7b3a6 Jul 28 17:09:16.637671 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:09:16.637605 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab5e312_2589_4188_b5d1_fe77447c7e52.slice/crio-b38ea2cb855bd09ef777bc545d4a2af299b6ab3fbd0d9dc6f694ad0c0f779e4d WatchSource:0}: Error finding container b38ea2cb855bd09ef777bc545d4a2af299b6ab3fbd0d9dc6f694ad0c0f779e4d: Status 404 returned error can't find the container with id b38ea2cb855bd09ef777bc545d4a2af299b6ab3fbd0d9dc6f694ad0c0f779e4d Jul 28 17:09:16.869761 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.869688 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:16.869761 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:16.869732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:16.869962 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.869839 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:16.869962 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.869857 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:16.869962 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.869909 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:09:17.869891978 +0000 UTC m=+35.059665363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:16.869962 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.869836 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:16.870226 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:16.870013 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:17.869995089 +0000 UTC m=+35.059768456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:17.333197 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.332576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:17.335668 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.335634 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-65cz5\"" Jul 28 17:09:17.347218 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.347088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:17.632295 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.631959 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4pld" event={"ID":"8ab5e312-2589-4188-b5d1-fe77447c7e52","Type":"ContainerStarted","Data":"b38ea2cb855bd09ef777bc545d4a2af299b6ab3fbd0d9dc6f694ad0c0f779e4d"} Jul 28 17:09:17.633423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.633378 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" event={"ID":"44b20fb3-a288-49a1-bc95-a87e57017d08","Type":"ContainerStarted","Data":"9c7179705bf9090fc3e6516bbe99c12122af69574892089a9c63de290ba7b3a6"} Jul 28 17:09:17.877520 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.877356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:17.877520 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:17.877446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:17.877759 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:17.877552 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:17.877759 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:17.877570 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:17.877759 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:17.877593 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:17.877759 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:17.877617 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:19.877600116 +0000 UTC m=+37.067373484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:17.877759 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:17.877646 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:09:19.877629019 +0000 UTC m=+37.067402388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:19.036620 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.036590 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc"] Jul 28 17:09:19.044303 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.044280 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.047643 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.047593 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 28 17:09:19.048260 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.048075 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 28 17:09:19.048733 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.048714 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n9frc\"" Jul 28 17:09:19.049639 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.049615 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc"] Jul 28 17:09:19.185803 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.185765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.185978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.185888 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19796901-f223-43c6-9e19-d37b707ca92e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.286973 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.286859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.286973 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.286918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19796901-f223-43c6-9e19-d37b707ca92e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.287366 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.287060 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:19.287366 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.287150 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:19.78712753 +0000 UTC m=+36.976900897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:19.287657 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.287626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19796901-f223-43c6-9e19-d37b707ca92e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.790660 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.790628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:19.790809 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.790747 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:19.790854 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.790812 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.790798522 +0000 UTC m=+37.980571885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:19.891406 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.891367 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:19.891534 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:19.891424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:19.891534 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.891505 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:19.891616 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.891561 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:23.891545461 +0000 UTC m=+41.081318824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:19.891616 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.891511 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:19.891616 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.891605 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:19.891714 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:19.891641 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:09:23.891631523 +0000 UTC m=+41.081404885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:20.800548 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:20.800509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:20.801065 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:20.800680 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:20.801065 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:20.800773 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.800754207 +0000 UTC m=+39.990527593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:21.513729 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.513707 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k6847"] Jul 28 17:09:21.515665 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:09:21.515643 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d690c0_0665_4922_9786_e2ab30ea50b9.slice/crio-ab83353a5e4e576fd871895d8ac13513f081a2cde13af1ed9c192c06ca3907aa WatchSource:0}: Error finding container ab83353a5e4e576fd871895d8ac13513f081a2cde13af1ed9c192c06ca3907aa: Status 404 returned error can't find the container with id ab83353a5e4e576fd871895d8ac13513f081a2cde13af1ed9c192c06ca3907aa Jul 28 17:09:21.644840 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.644814 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4pld" event={"ID":"8ab5e312-2589-4188-b5d1-fe77447c7e52","Type":"ContainerStarted","Data":"6ff0bb03d4ce93bf28cf396eb8fcda714087ade5f447a49e4bc343ef4bd475e8"} Jul 28 17:09:21.646441 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.646415 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" event={"ID":"44b20fb3-a288-49a1-bc95-a87e57017d08","Type":"ContainerStarted","Data":"f1adbfbfe972b328eb1718b2541f515493fcca81308ac7daf259f1c2840f3c5b"} Jul 28 17:09:21.648353 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.648086 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k6847" event={"ID":"d3d690c0-0665-4922-9786-e2ab30ea50b9","Type":"ContainerStarted","Data":"3a31d99197aaf7524e69cd6681a6d368ca9a6694268468624298b1a19c1d04b1"} Jul 28 17:09:21.648353 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.648128 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k6847" event={"ID":"d3d690c0-0665-4922-9786-e2ab30ea50b9","Type":"ContainerStarted","Data":"ab83353a5e4e576fd871895d8ac13513f081a2cde13af1ed9c192c06ca3907aa"} Jul 28 17:09:21.648353 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.648310 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:09:21.652676 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.650878 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerStarted","Data":"069f3849dcdf1e154853694334cc7b5740e15419e89830c9af4c1a7e068c4746"} Jul 28 17:09:21.661899 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.661858 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r4pld" podStartSLOduration=0.909767594 podStartE2EDuration="5.661844974s" totalImagesPullingTime="4.75207738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:21.661284458 +0000 UTC m=+38.851057844" watchObservedRunningTime="2026-07-28 17:09:21.661844974 +0000 UTC m=+38.851618392" Jul 28 17:09:21.699633 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.699587 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wzm7p" podStartSLOduration=27.936188713 podStartE2EDuration="32.699570761s" totalImagesPullingTime="4.763382048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:21.699056181 +0000 UTC m=+38.888829564" watchObservedRunningTime="2026-07-28 17:09:21.699570761 +0000 UTC m=+38.889344146" Jul 28 17:09:21.715357 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:21.715305 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k6847" podStartSLOduration=38.71528514 podStartE2EDuration="38.71528514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:21.714708681 +0000 UTC m=+38.904482066" watchObservedRunningTime="2026-07-28 17:09:21.71528514 +0000 UTC m=+38.905058527" Jul 28 17:09:22.655756 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:22.655676 2572 generic.go:356] "Generic (PLEG): container finished" podID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" containerID="069f3849dcdf1e154853694334cc7b5740e15419e89830c9af4c1a7e068c4746" exitCode=0 Jul 28 17:09:22.656116 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:22.655778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerDied","Data":"069f3849dcdf1e154853694334cc7b5740e15419e89830c9af4c1a7e068c4746"} Jul 28 17:09:22.815836 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:22.815806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:22.815975 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:22.815950 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:22.816027 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:22.816008 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.815989594 +0000 UTC m=+44.005762960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:23.661085 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:23.661056 2572 generic.go:356] "Generic (PLEG): container finished" podID="071211d6-06de-48c3-8f5e-fd7b5edc77d7" containerID="eebe2a1f28479a9b6290ce684bdcba3b0af4725b007fdac4df17363c3cc33166" exitCode=0 Jul 28 17:09:23.661453 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:23.661123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerDied","Data":"eebe2a1f28479a9b6290ce684bdcba3b0af4725b007fdac4df17363c3cc33166"} Jul 28 17:09:23.924912 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:23.924849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:23.925025 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:23.924917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:23.925025 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:23.924974 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:23.925025 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:23.924992 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:23.925025 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:23.924995 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:23.925171 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:23.925039 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:31.925026254 +0000 UTC m=+49.114799617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:23.925171 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:23.925053 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:09:31.925046766 +0000 UTC m=+49.114820128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:24.666954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:24.666917 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gljn7" event={"ID":"071211d6-06de-48c3-8f5e-fd7b5edc77d7","Type":"ContainerStarted","Data":"f55c98615e99eea4a48cd7e19726ed291384aaa266e0581e547dd945444febf5"} Jul 28 17:09:24.692090 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:24.692034 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gljn7" podStartSLOduration=4.934807035 podStartE2EDuration="41.69201948s" totalImagesPullingTime="36.757212445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 17:09:24.690414396 +0000 UTC m=+41.880187774" watchObservedRunningTime="2026-07-28 17:09:24.69201948 +0000 UTC m=+41.881792865" Jul 28 17:09:26.845014 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:26.844980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:26.845414 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:26.845124 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:26.845414 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:26.845185 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:34.845171667 +0000 UTC m=+52.034945030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:31.981312 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:31.981277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:31.981888 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:31.981324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:31.981888 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:31.981868 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:31.981888 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:31.981871 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:31.982091 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:31.981896 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:31.982091 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:31.981966 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:47.981947983 +0000 UTC m=+65.171721361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:31.982274 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:31.982256 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:09:47.982224869 +0000 UTC m=+65.171998251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:34.904102 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:34.904055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:34.904508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:34.904211 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:34.904508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:34.904273 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:50.904257658 +0000 UTC m=+68.094031021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:42.623498 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:42.623471 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpnss" Jul 28 17:09:47.987965 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:47.987915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:09:47.988508 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:47.988022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:09:47.988508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:47.988082 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:47.988508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:47.988099 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:09:47.988508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:47.988107 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:47.988508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:47.988162 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.9881485 +0000 UTC m=+97.177921863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:09:47.988508 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:47.988177 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.988171606 +0000 UTC m=+97.177944968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:09:48.089307 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.089276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:09:48.089476 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.089318 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:48.091532 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.091514 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:09:48.091635 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.091617 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:09:48.100312 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:48.100290 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 17:09:48.100410 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:48.100342 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:52.100327843 +0000 UTC m=+129.290101205 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : secret "metrics-daemon-secret" not found Jul 28 17:09:48.103294 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.103275 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d1534dc-38b2-4881-9fd1-80795b1e4b24-original-pull-secret\") pod \"global-pull-secret-syncer-nc9d4\" (UID: \"0d1534dc-38b2-4881-9fd1-80795b1e4b24\") " pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:48.143084 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.143064 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nc9d4" Jul 28 17:09:48.271737 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.271670 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nc9d4"] Jul 28 17:09:48.728280 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:48.728246 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nc9d4" event={"ID":"0d1534dc-38b2-4881-9fd1-80795b1e4b24","Type":"ContainerStarted","Data":"7b5e51ea0ccaffc96325063fce2ae2882963d5e5137fdd6bf933fe3d34620334"} Jul 28 17:09:50.912733 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:50.912696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:09:50.913195 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:50.912822 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:09:50.913195 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:09:50.912881 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.912864361 +0000 UTC m=+100.102637725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:09:52.740483 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:52.740450 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nc9d4" event={"ID":"0d1534dc-38b2-4881-9fd1-80795b1e4b24","Type":"ContainerStarted","Data":"25f55415c5d56d584c95d36fe66f3b054430a6636c382de9707dd853230d0d47"} Jul 28 17:09:52.755589 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:52.755542 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nc9d4" podStartSLOduration=66.227595811 podStartE2EDuration="1m9.755526523s" totalImagesPullingTime="3.527930712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:52.754927576 +0000 UTC m=+69.944700966" watchObservedRunningTime="2026-07-28 17:09:52.755526523 +0000 UTC m=+69.945299907" Jul 28 17:09:53.663323 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:09:53.663291 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k6847" Jul 28 17:10:20.018071 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:10:20.017953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:10:20.018071 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:10:20.017999 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:10:20.018662 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:20.018100 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:10:20.018662 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:20.018182 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.018165102 +0000 UTC m=+161.207938469 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:10:20.018662 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:20.018110 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:10:20.018662 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:20.018213 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:10:20.018662 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:20.018298 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.018283041 +0000 UTC m=+161.208056404 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:10:22.939935 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:10:22.939898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:10:22.940426 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:22.940066 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:10:22.940426 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:22.940148 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:26.940127522 +0000 UTC m=+164.129900885 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:10:52.138019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:10:52.137963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:10:52.138839 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:52.138155 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 17:10:52.138839 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:10:52.138257 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:54.138232367 +0000 UTC m=+251.328005729 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : secret "metrics-daemon-secret" not found Jul 28 17:11:19.168118 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:19.168078 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" podUID="bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" Jul 28 17:11:19.191421 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:19.191371 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6kxfc" podUID="49d74a24-e3fd-4b88-84eb-f2fa8df30516" Jul 28 17:11:19.348490 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:19.348456 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:11:19.951600 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:19.951574 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:11:19.951754 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:19.951574 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:11:22.054873 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:22.054832 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" podUID="19796901-f223-43c6-9e19-d37b707ca92e" Jul 28 17:11:22.959037 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:22.958993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:11:24.059347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:24.059312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:11:24.059807 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:24.059379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:11:24.059807 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:24.059499 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:11:24.059807 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:24.059523 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:11:24.059807 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:24.059499 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:11:24.059807 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:24.059586 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:13:26.059567654 +0000 UTC m=+283.249341034 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:11:24.059807 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:24.059600 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:26.059594399 +0000 UTC m=+283.249367761 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:11:26.977562 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:26.977528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:11:26.978028 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:26.977639 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:11:26.978028 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:11:26.977690 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:13:28.97767626 +0000 UTC m=+286.167449623 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:11:30.331719 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:11:30.331672 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:12:54.152229 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:12:54.152190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:12:54.152739 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:12:54.152344 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 17:12:54.152739 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:12:54.152441 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs podName:61deeab5-0b0e-4a34-80ea-e476ec574ee7 nodeName:}" failed. No retries permitted until 2026-07-28 17:14:56.15242409 +0000 UTC m=+373.342197452 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs") pod "network-metrics-daemon-m4kwh" (UID: "61deeab5-0b0e-4a34-80ea-e476ec574ee7") : secret "metrics-daemon-secret" not found Jul 28 17:13:12.881563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.881482 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn"] Jul 28 17:13:12.884417 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.884384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:12.886488 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.886459 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 17:13:12.886593 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.886529 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 17:13:12.887240 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.887221 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 17:13:12.887605 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.887206 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:13:12.887605 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.887244 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wtfwv\"" Jul 28 17:13:12.887782 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.887709 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz"] Jul 28 17:13:12.890204 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.890186 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m"] Jul 28 17:13:12.890384 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.890362 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:12.893586 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.893566 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tmsz5\"" Jul 28 17:13:12.893899 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.893877 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 17:13:12.894282 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.894263 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 17:13:12.896157 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.894503 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 17:13:12.896157 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.895787 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 17:13:12.896157 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.896045 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6"] Jul 28 17:13:12.896356 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.896160 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:12.898621 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.898583 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:13:12.898727 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.898640 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5ckvf\"" Jul 28 17:13:12.898849 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.898823 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 17:13:12.899533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.899380 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 17:13:12.899954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.899934 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 17:13:12.900245 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.900226 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn"] Jul 28 17:13:12.900337 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.900260 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6"] Jul 28 17:13:12.900421 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.900348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" Jul 28 17:13:12.902541 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.902341 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 17:13:12.903161 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.903111 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m"] Jul 28 17:13:12.903693 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.903445 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:13:12.903693 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.903501 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tmh47\"" Jul 28 17:13:12.904204 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.904187 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz"] Jul 28 17:13:12.984121 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.984101 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kgzlw"] Jul 28 17:13:12.986752 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.986739 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68ccfc8d67-d2tcw"] Jul 28 17:13:12.986828 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.986813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:12.989061 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.989046 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 17:13:12.989399 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.989367 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hjpxn\"" Jul 28 17:13:12.989399 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.989398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:12.989553 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.989369 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 17:13:12.989803 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.989787 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 17:13:12.990251 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990237 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 17:13:12.990537 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:12.990585 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990564 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff59\" (UniqueName: \"kubernetes.io/projected/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-kube-api-access-vff59\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:12.990631 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990587 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-config\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:12.990631 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990604 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:12.990735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990675 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxdv\" (UniqueName: \"kubernetes.io/projected/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-kube-api-access-grxdv\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:12.990735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.990727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:12.991918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.991898 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 17:13:12.992003 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.991885 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 17:13:12.992116 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.992099 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 17:13:12.992234 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.992220 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 17:13:12.992320 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.992306 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 17:13:12.992543 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.992528 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 17:13:12.992607 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.992569 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xbrdv\"" Jul 28 17:13:12.997597 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.997580 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kgzlw"] Jul 28 17:13:12.998452 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.998430 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 17:13:12.999995 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:12.999977 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68ccfc8d67-d2tcw"] Jul 28 17:13:13.091735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.091735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091738 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.091874 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kccj8\" (UniqueName: \"kubernetes.io/projected/9cf606e8-a632-48cc-a9b1-01bd9200de94-kube-api-access-kccj8\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.091874 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cf606e8-a632-48cc-a9b1-01bd9200de94-serving-cert\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.091874 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091800 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1bb1ced-d125-4861-b579-954c9e7558ba-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.091874 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-default-certificate\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.091874 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxdv\" (UniqueName: \"kubernetes.io/projected/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-kube-api-access-grxdv\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:13.091874 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf606e8-a632-48cc-a9b1-01bd9200de94-service-ca-bundle\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.092091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1bb1ced-d125-4861-b579-954c9e7558ba-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.092091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091924 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtqh\" (UniqueName: \"kubernetes.io/projected/cbf0c30a-2f8a-4c94-a25d-55140d894389-kube-api-access-7dtqh\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.092091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4q8\" (UniqueName: \"kubernetes.io/projected/a1bb1ced-d125-4861-b579-954c9e7558ba-kube-api-access-7w4q8\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.092091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.091980 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf606e8-a632-48cc-a9b1-01bd9200de94-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.092091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff59\" (UniqueName: \"kubernetes.io/projected/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-kube-api-access-vff59\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.092259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cf606e8-a632-48cc-a9b1-01bd9200de94-tmp\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.092259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.092259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:13.092259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmcg\" (UniqueName: \"kubernetes.io/projected/15f02567-4b4a-4ac2-9b2b-844e91578ebb-kube-api-access-vqmcg\") pod \"volume-data-source-validator-549c6d54cf-x4bv6\" (UID: \"15f02567-4b4a-4ac2-9b2b-844e91578ebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" Jul 28 17:13:13.092259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9cf606e8-a632-48cc-a9b1-01bd9200de94-snapshots\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.092259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:13.092493 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.092272 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:13.092493 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-config\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.092493 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-stats-auth\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.092493 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.092344 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls podName:bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:13.592324917 +0000 UTC m=+270.782098280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bp2dz" (UID: "bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:13.092840 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.092821 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-config\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.093015 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.093001 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:13.094003 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.093981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.102033 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.102012 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff59\" (UniqueName: \"kubernetes.io/projected/0578c2fb-d319-4d60-a4b7-465db4a7a3e4-kube-api-access-vff59\") pod \"service-ca-operator-5d6c75f84f-km5nn\" (UID: \"0578c2fb-d319-4d60-a4b7-465db4a7a3e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.102300 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.102283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxdv\" (UniqueName: \"kubernetes.io/projected/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-kube-api-access-grxdv\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:13.193332 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-stats-auth\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.193332 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kccj8\" (UniqueName: \"kubernetes.io/projected/9cf606e8-a632-48cc-a9b1-01bd9200de94-kube-api-access-kccj8\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cf606e8-a632-48cc-a9b1-01bd9200de94-serving-cert\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1bb1ced-d125-4861-b579-954c9e7558ba-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-default-certificate\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.193432 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf606e8-a632-48cc-a9b1-01bd9200de94-service-ca-bundle\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1bb1ced-d125-4861-b579-954c9e7558ba-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.193486 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.193488 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:13.693471499 +0000 UTC m=+270.883244866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : secret "router-metrics-certs-default" not found Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtqh\" (UniqueName: \"kubernetes.io/projected/cbf0c30a-2f8a-4c94-a25d-55140d894389-kube-api-access-7dtqh\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4q8\" (UniqueName: \"kubernetes.io/projected/a1bb1ced-d125-4861-b579-954c9e7558ba-kube-api-access-7w4q8\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf606e8-a632-48cc-a9b1-01bd9200de94-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cf606e8-a632-48cc-a9b1-01bd9200de94-tmp\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmcg\" (UniqueName: \"kubernetes.io/projected/15f02567-4b4a-4ac2-9b2b-844e91578ebb-kube-api-access-vqmcg\") pod \"volume-data-source-validator-549c6d54cf-x4bv6\" (UID: \"15f02567-4b4a-4ac2-9b2b-844e91578ebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" Jul 28 17:13:13.193892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.193746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9cf606e8-a632-48cc-a9b1-01bd9200de94-snapshots\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.194228 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.194088 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1bb1ced-d125-4861-b579-954c9e7558ba-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.194499 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.194479 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:13.694461951 +0000 UTC m=+270.884235330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : configmap references non-existent config key: service-ca.crt Jul 28 17:13:13.194598 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.194578 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9cf606e8-a632-48cc-a9b1-01bd9200de94-snapshots\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.194813 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.194792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cf606e8-a632-48cc-a9b1-01bd9200de94-tmp\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.195092 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.195065 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf606e8-a632-48cc-a9b1-01bd9200de94-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.195316 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.195295 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf606e8-a632-48cc-a9b1-01bd9200de94-service-ca-bundle\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.195835 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.195814 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1bb1ced-d125-4861-b579-954c9e7558ba-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.195906 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.195892 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-default-certificate\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.195958 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.195942 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-stats-auth\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.196023 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.196006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9cf606e8-a632-48cc-a9b1-01bd9200de94-serving-cert\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.198887 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.198864 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" Jul 28 17:13:13.204532 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.204506 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kccj8\" (UniqueName: \"kubernetes.io/projected/9cf606e8-a632-48cc-a9b1-01bd9200de94-kube-api-access-kccj8\") pod \"insights-operator-5885fdb759-kgzlw\" (UID: \"9cf606e8-a632-48cc-a9b1-01bd9200de94\") " pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.205064 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.205045 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4q8\" (UniqueName: \"kubernetes.io/projected/a1bb1ced-d125-4861-b579-954c9e7558ba-kube-api-access-7w4q8\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m\" (UID: \"a1bb1ced-d125-4861-b579-954c9e7558ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.205311 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.205294 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmcg\" (UniqueName: \"kubernetes.io/projected/15f02567-4b4a-4ac2-9b2b-844e91578ebb-kube-api-access-vqmcg\") pod \"volume-data-source-validator-549c6d54cf-x4bv6\" (UID: \"15f02567-4b4a-4ac2-9b2b-844e91578ebb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" Jul 28 17:13:13.205371 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.205315 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtqh\" (UniqueName: \"kubernetes.io/projected/cbf0c30a-2f8a-4c94-a25d-55140d894389-kube-api-access-7dtqh\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.215241 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.215223 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" Jul 28 17:13:13.220038 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.220019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" Jul 28 17:13:13.296592 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.296567 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" Jul 28 17:13:13.346881 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.346852 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn"] Jul 28 17:13:13.351221 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:13.351190 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0578c2fb_d319_4d60_a4b7_465db4a7a3e4.slice/crio-d06f56b6ed93e4af82e4d75f11595457352394c6e2675b8222189dd67fd37c3b WatchSource:0}: Error finding container d06f56b6ed93e4af82e4d75f11595457352394c6e2675b8222189dd67fd37c3b: Status 404 returned error can't find the container with id d06f56b6ed93e4af82e4d75f11595457352394c6e2675b8222189dd67fd37c3b Jul 28 17:13:13.360489 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.360469 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m"] Jul 28 17:13:13.364623 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:13.364602 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bb1ced_d125_4861_b579_954c9e7558ba.slice/crio-e6d6d628f1786d0dbaffe1a30e394fb6982f2c10c5cad6d235d4e1ae732e8543 WatchSource:0}: Error finding container e6d6d628f1786d0dbaffe1a30e394fb6982f2c10c5cad6d235d4e1ae732e8543: Status 404 returned error can't find the container with id e6d6d628f1786d0dbaffe1a30e394fb6982f2c10c5cad6d235d4e1ae732e8543 Jul 28 17:13:13.380812 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.380788 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6"] Jul 28 17:13:13.383546 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:13.383522 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f02567_4b4a_4ac2_9b2b_844e91578ebb.slice/crio-70fad722c56eaa6b8ccecd0ba0ab5347de9c5f26028d1b5d2eeb0d69eceb4009 WatchSource:0}: Error finding container 70fad722c56eaa6b8ccecd0ba0ab5347de9c5f26028d1b5d2eeb0d69eceb4009: Status 404 returned error can't find the container with id 70fad722c56eaa6b8ccecd0ba0ab5347de9c5f26028d1b5d2eeb0d69eceb4009 Jul 28 17:13:13.440483 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.440373 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kgzlw"] Jul 28 17:13:13.442702 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:13.442679 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf606e8_a632_48cc_a9b1_01bd9200de94.slice/crio-097b29572ebd5f57de24347239acf181f0a4103e0d2db1122d2ab43fe5bd7131 WatchSource:0}: Error finding container 097b29572ebd5f57de24347239acf181f0a4103e0d2db1122d2ab43fe5bd7131: Status 404 returned error can't find the container with id 097b29572ebd5f57de24347239acf181f0a4103e0d2db1122d2ab43fe5bd7131 Jul 28 17:13:13.597483 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.597456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:13.597636 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.597591 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:13.597685 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.597650 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls podName:bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:14.597635973 +0000 UTC m=+271.787409339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bp2dz" (UID: "bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:13.698304 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.698244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.698438 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:13.698306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:13.698438 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.698405 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:14.698375592 +0000 UTC m=+271.888148954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : configmap references non-existent config key: service-ca.crt Jul 28 17:13:13.698438 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.698429 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 17:13:13.698606 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:13.698470 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:14.698459156 +0000 UTC m=+271.888232518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : secret "router-metrics-certs-default" not found Jul 28 17:13:14.227945 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.227863 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" event={"ID":"a1bb1ced-d125-4861-b579-954c9e7558ba","Type":"ContainerStarted","Data":"e6d6d628f1786d0dbaffe1a30e394fb6982f2c10c5cad6d235d4e1ae732e8543"} Jul 28 17:13:14.229443 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.229410 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" event={"ID":"0578c2fb-d319-4d60-a4b7-465db4a7a3e4","Type":"ContainerStarted","Data":"d06f56b6ed93e4af82e4d75f11595457352394c6e2675b8222189dd67fd37c3b"} Jul 28 17:13:14.231197 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.231171 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" event={"ID":"9cf606e8-a632-48cc-a9b1-01bd9200de94","Type":"ContainerStarted","Data":"097b29572ebd5f57de24347239acf181f0a4103e0d2db1122d2ab43fe5bd7131"} Jul 28 17:13:14.232888 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.232839 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" event={"ID":"15f02567-4b4a-4ac2-9b2b-844e91578ebb","Type":"ContainerStarted","Data":"70fad722c56eaa6b8ccecd0ba0ab5347de9c5f26028d1b5d2eeb0d69eceb4009"} Jul 28 17:13:14.607361 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.606735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:14.607361 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:14.606927 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:14.607361 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:14.606991 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls podName:bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:16.606972283 +0000 UTC m=+273.796745669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bp2dz" (UID: "bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:14.707570 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.707533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:14.707735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:14.707636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:14.707835 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:14.707816 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:16.707788999 +0000 UTC m=+273.897562370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : configmap references non-existent config key: service-ca.crt Jul 28 17:13:14.707908 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:14.707822 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 17:13:14.707908 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:14.707871 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:16.707861487 +0000 UTC m=+273.897634853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : secret "router-metrics-certs-default" not found Jul 28 17:13:16.624915 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:16.624885 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:16.625357 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:16.625044 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:16.625357 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:16.625276 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls podName:bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:20.625257255 +0000 UTC m=+277.815030635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bp2dz" (UID: "bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:16.725630 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:16.725559 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:16.725630 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:16.725625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:16.725820 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:16.725750 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:20.725730055 +0000 UTC m=+277.915503422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : configmap references non-existent config key: service-ca.crt Jul 28 17:13:16.725930 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:16.725909 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 17:13:16.726011 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:16.726001 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:20.725985224 +0000 UTC m=+277.915758590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : secret "router-metrics-certs-default" not found Jul 28 17:13:17.244547 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.244513 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" event={"ID":"0578c2fb-d319-4d60-a4b7-465db4a7a3e4","Type":"ContainerStarted","Data":"4328c95c4473078177b60ffc7f7755e69e454ec4baffe0267d25318a0e952a28"} Jul 28 17:13:17.246174 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.246148 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" event={"ID":"9cf606e8-a632-48cc-a9b1-01bd9200de94","Type":"ContainerStarted","Data":"6f440cac3ef8f6f5ca4fbe1ccc8dceeeb733f3736932556c19bb8e68bbcb0e4f"} Jul 28 17:13:17.247743 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.247717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" event={"ID":"15f02567-4b4a-4ac2-9b2b-844e91578ebb","Type":"ContainerStarted","Data":"b2d5ccfd62c62826388241a1b72a9787972240fa4c639fc4239e118dec1e64e4"} Jul 28 17:13:17.249329 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.249306 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" event={"ID":"a1bb1ced-d125-4861-b579-954c9e7558ba","Type":"ContainerStarted","Data":"4cc4d39e515a21dac5d996b9273cd23d6bf8b59d81335a6c0461b2fc37375aa0"} Jul 28 17:13:17.262180 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.262120 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" podStartSLOduration=2.145641869 podStartE2EDuration="5.262107958s" totalImagesPullingTime="3.116466089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:17.261543793 +0000 UTC m=+274.451317169" watchObservedRunningTime="2026-07-28 17:13:17.262107958 +0000 UTC m=+274.451881342" Jul 28 17:13:17.277198 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.277135 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" podStartSLOduration=2.179304269 podStartE2EDuration="5.277117599s" totalImagesPullingTime="3.09781333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:17.276670785 +0000 UTC m=+274.466444171" watchObservedRunningTime="2026-07-28 17:13:17.277117599 +0000 UTC m=+274.466890988" Jul 28 17:13:17.294832 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.294790 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" podStartSLOduration=2.275553931 podStartE2EDuration="5.294772989s" totalImagesPullingTime="3.019219058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:17.293288033 +0000 UTC m=+274.483061548" watchObservedRunningTime="2026-07-28 17:13:17.294772989 +0000 UTC m=+274.484546374" Jul 28 17:13:17.309767 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:17.309719 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-x4bv6" podStartSLOduration=2.230977414 podStartE2EDuration="5.309704426s" totalImagesPullingTime="3.078727012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:17.308451193 +0000 UTC m=+274.498224580" watchObservedRunningTime="2026-07-28 17:13:17.309704426 +0000 UTC m=+274.499477812" Jul 28 17:13:18.123518 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.123482 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-djchv"] Jul 28 17:13:18.127793 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.127764 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.129913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.129878 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 17:13:18.129913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.129878 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 17:13:18.130235 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.130214 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hktz8\"" Jul 28 17:13:18.135602 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.135578 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.135725 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.135622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-data-volume\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.135725 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.135709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvgq\" (UniqueName: \"kubernetes.io/projected/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-api-access-vxvgq\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.135833 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.135818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.135939 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.135913 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-crio-socket\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.137023 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.136995 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-djchv"] Jul 28 17:13:18.236959 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.236930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvgq\" (UniqueName: \"kubernetes.io/projected/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-api-access-vxvgq\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.237067 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.236995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.237125 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.237071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-crio-socket\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.237125 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.237102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.237244 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.237130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-data-volume\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.237244 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.237139 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:13:18.237244 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.237152 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-crio-socket\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.237244 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.237212 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:18.73719038 +0000 UTC m=+275.926963764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : secret "insights-runtime-extractor-tls" not found Jul 28 17:13:18.237244 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.237217 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:13:18.237531 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.237257 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:18.737245304 +0000 UTC m=+275.927018679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : configmap "kube-rbac-proxy" not found Jul 28 17:13:18.237583 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.237526 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-data-volume\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.246996 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.246977 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvgq\" (UniqueName: \"kubernetes.io/projected/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-api-access-vxvgq\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.741444 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.741403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.741634 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:18.741496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:18.741634 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.741546 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:13:18.741634 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.741583 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:13:18.741634 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.741613 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:19.741597097 +0000 UTC m=+276.931370459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : secret "insights-runtime-extractor-tls" not found Jul 28 17:13:18.741634 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:18.741637 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:19.741620095 +0000 UTC m=+276.931393458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : configmap "kube-rbac-proxy" not found Jul 28 17:13:19.748018 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:19.747984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:19.748433 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:19.748072 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:19.748433 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:19.748095 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:13:19.748433 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:19.748165 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:21.748148254 +0000 UTC m=+278.937921617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : configmap "kube-rbac-proxy" not found Jul 28 17:13:19.748433 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:19.748189 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:13:19.748433 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:19.748236 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:21.748220139 +0000 UTC m=+278.937993502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : secret "insights-runtime-extractor-tls" not found Jul 28 17:13:20.654360 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:20.654328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:20.654532 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:20.654481 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:20.654580 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:20.654542 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls podName:bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:28.654526919 +0000 UTC m=+285.844300281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bp2dz" (UID: "bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:20.755214 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:20.755176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:20.755610 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:20.755268 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:20.755610 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:20.755314 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 17:13:20.755610 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:20.755373 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:28.755359908 +0000 UTC m=+285.945133271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : secret "router-metrics-certs-default" not found Jul 28 17:13:20.755610 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:20.755404 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:28.755380823 +0000 UTC m=+285.945154186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : configmap references non-existent config key: service-ca.crt Jul 28 17:13:20.912813 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:20.912744 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npp2t_aca3ad49-96eb-4f52-b11e-4bea99d26f5d/dns-node-resolver/0.log" Jul 28 17:13:21.763066 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:21.763033 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:21.763449 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:21.763103 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:21.763449 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:21.763178 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:13:21.763449 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:21.763193 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:13:21.763449 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:21.763242 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:25.763228397 +0000 UTC m=+282.953001760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : secret "insights-runtime-extractor-tls" not found Jul 28 17:13:21.763449 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:21.763274 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:25.763254196 +0000 UTC m=+282.953027562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : configmap "kube-rbac-proxy" not found Jul 28 17:13:21.912746 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:21.912724 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tdhg2_156ff3fc-7c3f-49d5-a362-949d57777cf1/node-ca/0.log" Jul 28 17:13:22.712255 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:22.712230 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4pld_8ab5e312-2589-4188-b5d1-fe77447c7e52/serve-healthcheck-canary/0.log" Jul 28 17:13:22.953043 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:22.953001 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" podUID="bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" Jul 28 17:13:22.953043 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:22.953001 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6kxfc" podUID="49d74a24-e3fd-4b88-84eb-f2fa8df30516" Jul 28 17:13:23.267102 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:23.267069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:13:23.267273 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:23.267069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:13:23.515844 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:23.515812 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m_a1bb1ced-d125-4861-b579-954c9e7558ba/kube-storage-version-migrator-operator/0.log" Jul 28 17:13:25.796764 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:25.796704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:25.797224 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:25.796871 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:13:25.797224 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:25.796927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:25.797224 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:25.796944 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:33.796925929 +0000 UTC m=+290.986699310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : secret "insights-runtime-extractor-tls" not found Jul 28 17:13:25.797224 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:25.796980 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:13:25.797224 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:25.797015 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm podName:cd03507b-8eb3-40bc-ad2e-8c1c7c05473a nodeName:}" failed. No retries permitted until 2026-07-28 17:13:33.797005086 +0000 UTC m=+290.986778517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-djchv" (UID: "cd03507b-8eb3-40bc-ad2e-8c1c7c05473a") : configmap "kube-rbac-proxy" not found Jul 28 17:13:25.960286 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:25.960240 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" podUID="19796901-f223-43c6-9e19-d37b707ca92e" Jul 28 17:13:26.099354 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:26.099266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") pod \"image-registry-65d68454b6-wdf5b\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:13:26.099543 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:26.099366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:13:26.099543 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:26.099455 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:13:26.099543 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:26.099481 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d68454b6-wdf5b: secret "image-registry-tls" not found Jul 28 17:13:26.099543 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:26.099542 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls podName:bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab nodeName:}" failed. No retries permitted until 2026-07-28 17:15:28.099521627 +0000 UTC m=+405.289294990 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls") pod "image-registry-65d68454b6-wdf5b" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab") : secret "image-registry-tls" not found Jul 28 17:13:26.099752 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:26.099463 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:13:26.099752 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:26.099595 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls podName:49d74a24-e3fd-4b88-84eb-f2fa8df30516 nodeName:}" failed. No retries permitted until 2026-07-28 17:15:28.099584471 +0000 UTC m=+405.289357834 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls") pod "dns-default-6kxfc" (UID: "49d74a24-e3fd-4b88-84eb-f2fa8df30516") : secret "dns-default-metrics-tls" not found Jul 28 17:13:26.275347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:26.275314 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:13:28.721575 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:28.721527 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:28.721953 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:28.721681 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:28.721953 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:28.721763 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls podName:bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:44.721747288 +0000 UTC m=+301.911520651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bp2dz" (UID: "bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8") : secret "cluster-monitoring-operator-tls" not found Jul 28 17:13:28.822712 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:28.822679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:28.822850 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:28.822758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:28.822850 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:28.822840 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle podName:cbf0c30a-2f8a-4c94-a25d-55140d894389 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:44.822823148 +0000 UTC m=+302.012596511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle") pod "router-default-68ccfc8d67-d2tcw" (UID: "cbf0c30a-2f8a-4c94-a25d-55140d894389") : configmap references non-existent config key: service-ca.crt Jul 28 17:13:28.825062 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:28.825036 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf0c30a-2f8a-4c94-a25d-55140d894389-metrics-certs\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:29.023783 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:29.023704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:13:29.023922 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:29.023821 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 17:13:29.023922 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:29.023875 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert podName:19796901-f223-43c6-9e19-d37b707ca92e nodeName:}" failed. No retries permitted until 2026-07-28 17:15:31.023862136 +0000 UTC m=+408.213635499 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5j6qc" (UID: "19796901-f223-43c6-9e19-d37b707ca92e") : secret "networking-console-plugin-cert" not found Jul 28 17:13:33.332816 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:33.332782 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-m4kwh" podUID="61deeab5-0b0e-4a34-80ea-e476ec574ee7" Jul 28 17:13:33.862168 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:33.862131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:33.862342 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:33.862215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:33.862739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:33.862703 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:33.864567 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:33.864538 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd03507b-8eb3-40bc-ad2e-8c1c7c05473a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-djchv\" (UID: \"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a\") " pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:34.038138 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:34.038119 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-djchv" Jul 28 17:13:34.159741 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:34.159695 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-djchv"] Jul 28 17:13:34.164512 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:34.164488 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd03507b_8eb3_40bc_ad2e_8c1c7c05473a.slice/crio-90746f1581be8ae2ecf1724a98fc2243d8f87829f8576f57b3895f5862f9f250 WatchSource:0}: Error finding container 90746f1581be8ae2ecf1724a98fc2243d8f87829f8576f57b3895f5862f9f250: Status 404 returned error can't find the container with id 90746f1581be8ae2ecf1724a98fc2243d8f87829f8576f57b3895f5862f9f250 Jul 28 17:13:34.300178 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:34.300142 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-djchv" event={"ID":"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a","Type":"ContainerStarted","Data":"90746f1581be8ae2ecf1724a98fc2243d8f87829f8576f57b3895f5862f9f250"} Jul 28 17:13:39.317654 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:39.317621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-djchv" event={"ID":"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a","Type":"ContainerStarted","Data":"433a7b73a1c1109971a9ef9d1d7e9ae291d351128e6f5ee27faba02b14317619"} Jul 28 17:13:43.223611 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:43.223580 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 17:13:43.336080 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:43.335952 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-djchv" event={"ID":"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a","Type":"ContainerStarted","Data":"642efa1c4c2bd7ef6e5099bb7419b753a595b4dd377643cb6428f124909a8911"} Jul 28 17:13:44.753370 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:44.753326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:44.756031 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:44.756006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bp2dz\" (UID: \"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:44.854479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:44.854438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:44.855023 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:44.854997 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf0c30a-2f8a-4c94-a25d-55140d894389-service-ca-bundle\") pod \"router-default-68ccfc8d67-d2tcw\" (UID: \"cbf0c30a-2f8a-4c94-a25d-55140d894389\") " pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:45.011527 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:45.011449 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tmsz5\"" Jul 28 17:13:45.019686 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:45.019662 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" Jul 28 17:13:45.104448 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:45.104410 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xbrdv\"" Jul 28 17:13:45.112298 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:45.112274 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:46.188091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.188064 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz"] Jul 28 17:13:46.190365 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:46.190343 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0e9bc9_8193_4d05_9bba_6b1f5d5f70d8.slice/crio-c1e8a0e6be9bb1a20278fa85e7810d9994e5439621413d2e40d7c28def56c293 WatchSource:0}: Error finding container c1e8a0e6be9bb1a20278fa85e7810d9994e5439621413d2e40d7c28def56c293: Status 404 returned error can't find the container with id c1e8a0e6be9bb1a20278fa85e7810d9994e5439621413d2e40d7c28def56c293 Jul 28 17:13:46.192277 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.192262 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:13:46.204568 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.204549 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68ccfc8d67-d2tcw"] Jul 28 17:13:46.206307 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:46.206284 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf0c30a_2f8a_4c94_a25d_55140d894389.slice/crio-bbbd892ed84b5d4dacd8c90ddc67b1e9440feae786f98b0894eb3ef64d9437a5 WatchSource:0}: Error finding container bbbd892ed84b5d4dacd8c90ddc67b1e9440feae786f98b0894eb3ef64d9437a5: Status 404 returned error can't find the container with id bbbd892ed84b5d4dacd8c90ddc67b1e9440feae786f98b0894eb3ef64d9437a5 Jul 28 17:13:46.345383 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.345311 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" event={"ID":"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8","Type":"ContainerStarted","Data":"c1e8a0e6be9bb1a20278fa85e7810d9994e5439621413d2e40d7c28def56c293"} Jul 28 17:13:46.347275 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.347253 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-djchv" event={"ID":"cd03507b-8eb3-40bc-ad2e-8c1c7c05473a","Type":"ContainerStarted","Data":"83085ee7d060af93490f9d99483a49686c5d3188e1b8a090b5831d3532d3193c"} Jul 28 17:13:46.348648 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.348627 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" event={"ID":"cbf0c30a-2f8a-4c94-a25d-55140d894389","Type":"ContainerStarted","Data":"3b474bc8711555f9a17f195a857ec722c7f4f34002c3bdacce6a499bfbcbcb6e"} Jul 28 17:13:46.348736 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.348662 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" event={"ID":"cbf0c30a-2f8a-4c94-a25d-55140d894389","Type":"ContainerStarted","Data":"bbbd892ed84b5d4dacd8c90ddc67b1e9440feae786f98b0894eb3ef64d9437a5"} Jul 28 17:13:46.368403 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.368355 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-djchv" podStartSLOduration=16.526455646 podStartE2EDuration="28.368340844s" totalImagesPullingTime="11.841885198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:13:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:46.3673389 +0000 UTC m=+303.557112284" watchObservedRunningTime="2026-07-28 17:13:46.368340844 +0000 UTC m=+303.558114229" Jul 28 17:13:46.397246 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:46.397204 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" podStartSLOduration=34.39718486 podStartE2EDuration="34.39718486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:46.396705141 +0000 UTC m=+303.586478526" watchObservedRunningTime="2026-07-28 17:13:46.39718486 +0000 UTC m=+303.586958245" Jul 28 17:13:47.112764 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:47.112735 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:47.115606 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:47.115583 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:47.353290 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:47.353232 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:47.354587 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:47.354563 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68ccfc8d67-d2tcw" Jul 28 17:13:48.332239 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:48.332214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:13:48.357332 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:48.357311 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" event={"ID":"bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8","Type":"ContainerStarted","Data":"8ecc23aa294656eb9c8f43091f1ed27a9f0cdcee3f86ca4fcbca620e5e960cdd"} Jul 28 17:13:49.380105 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:49.380057 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bp2dz" podStartSLOduration=35.276136434 podStartE2EDuration="37.380043068s" totalImagesPullingTime="2.103906634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:49.378325215 +0000 UTC m=+306.568098612" watchObservedRunningTime="2026-07-28 17:13:49.380043068 +0000 UTC m=+306.569816450" Jul 28 17:13:52.891597 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.891566 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-b8l29"] Jul 28 17:13:52.894064 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.894047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:52.896977 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.896956 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 17:13:52.897100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.897001 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h8jzf\"" Jul 28 17:13:52.897100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.897055 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 17:13:52.897239 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.897093 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 17:13:52.903804 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:52.903780 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-b8l29"] Jul 28 17:13:53.020781 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.020752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bd52883-679e-45e7-91d6-164cc3d9eb21-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.020913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.020803 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bd52883-679e-45e7-91d6-164cc3d9eb21-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.020913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.020874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bd52883-679e-45e7-91d6-164cc3d9eb21-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.021053 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.020932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44hx8\" (UniqueName: \"kubernetes.io/projected/3bd52883-679e-45e7-91d6-164cc3d9eb21-kube-api-access-44hx8\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.121224 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.121187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bd52883-679e-45e7-91d6-164cc3d9eb21-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.121313 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.121239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bd52883-679e-45e7-91d6-164cc3d9eb21-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.121313 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.121267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bd52883-679e-45e7-91d6-164cc3d9eb21-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.121313 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.121291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44hx8\" (UniqueName: \"kubernetes.io/projected/3bd52883-679e-45e7-91d6-164cc3d9eb21-kube-api-access-44hx8\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.121971 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.121951 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bd52883-679e-45e7-91d6-164cc3d9eb21-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.123658 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.123636 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bd52883-679e-45e7-91d6-164cc3d9eb21-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.123812 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.123792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bd52883-679e-45e7-91d6-164cc3d9eb21-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.132404 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.132372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44hx8\" (UniqueName: \"kubernetes.io/projected/3bd52883-679e-45e7-91d6-164cc3d9eb21-kube-api-access-44hx8\") pod \"prometheus-operator-84d69f794d-b8l29\" (UID: \"3bd52883-679e-45e7-91d6-164cc3d9eb21\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.203298 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.203255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" Jul 28 17:13:53.324145 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.324084 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-b8l29"] Jul 28 17:13:53.326615 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:53.326589 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd52883_679e_45e7_91d6_164cc3d9eb21.slice/crio-15ad713bc530be4ab80ea45d0223b4b6aeb080d0023a1d9f8c93aee7e76531a1 WatchSource:0}: Error finding container 15ad713bc530be4ab80ea45d0223b4b6aeb080d0023a1d9f8c93aee7e76531a1: Status 404 returned error can't find the container with id 15ad713bc530be4ab80ea45d0223b4b6aeb080d0023a1d9f8c93aee7e76531a1 Jul 28 17:13:53.374561 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:53.374535 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" event={"ID":"3bd52883-679e-45e7-91d6-164cc3d9eb21","Type":"ContainerStarted","Data":"15ad713bc530be4ab80ea45d0223b4b6aeb080d0023a1d9f8c93aee7e76531a1"} Jul 28 17:13:55.381921 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:55.381889 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" event={"ID":"3bd52883-679e-45e7-91d6-164cc3d9eb21","Type":"ContainerStarted","Data":"e13d1866604d6d00149b3b04c1cb3b9f722858df8a00fd6b1f6bd5974f8302f7"} Jul 28 17:13:55.382312 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:55.381938 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" event={"ID":"3bd52883-679e-45e7-91d6-164cc3d9eb21","Type":"ContainerStarted","Data":"2138029fa98546bd035fac8da1d733a081bce170dd9f2eb42764850dd2d9e4f7"} Jul 28 17:13:55.409981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:55.409935 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-b8l29" podStartSLOduration=2.377599779 podStartE2EDuration="3.409922037s" totalImagesPullingTime="1.032322258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:55.409725399 +0000 UTC m=+312.599498784" watchObservedRunningTime="2026-07-28 17:13:55.409922037 +0000 UTC m=+312.599695421" Jul 28 17:13:57.392461 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.392429 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm"] Jul 28 17:13:57.395551 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.395533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.397665 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.397647 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kj748\"" Jul 28 17:13:57.398096 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.398083 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 17:13:57.398096 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.398094 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:13:57.403632 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.403612 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm"] Jul 28 17:13:57.406501 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.406484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.410770 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.410751 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dqcsj\"" Jul 28 17:13:57.411320 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.411305 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 17:13:57.411657 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.411639 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 17:13:57.411754 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.411674 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:13:57.411928 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.411909 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm"] Jul 28 17:13:57.423063 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.423040 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm"] Jul 28 17:13:57.431776 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.431756 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4jhlt"] Jul 28 17:13:57.434656 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.434642 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.436794 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.436776 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 17:13:57.437204 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.437183 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 17:13:57.437253 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.437218 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zd2gj\"" Jul 28 17:13:57.437537 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.437523 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 17:13:57.555576 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.555682 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.555682 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-tls\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.555784 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10e0eb75-b3c8-428d-92d6-f27dc9829f89-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.555784 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555732 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-wtmp\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.555784 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7nh\" (UniqueName: \"kubernetes.io/projected/10e0eb75-b3c8-428d-92d6-f27dc9829f89-kube-api-access-qq7nh\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.555784 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.555983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-sys\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.555983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.555983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-root\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.555983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-textfile\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.555983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.555983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.555964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84ecc9cb-5524-46d0-b51c-31be4077f6f9-metrics-client-ca\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.556192 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.556002 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.556192 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.556029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86k4\" (UniqueName: \"kubernetes.io/projected/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-api-access-d86k4\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.556192 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.556070 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.556192 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.556107 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.556192 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.556130 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfw4\" (UniqueName: \"kubernetes.io/projected/84ecc9cb-5524-46d0-b51c-31be4077f6f9-kube-api-access-4qfw4\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.556192 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.556162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.656833 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.656766 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfw4\" (UniqueName: \"kubernetes.io/projected/84ecc9cb-5524-46d0-b51c-31be4077f6f9-kube-api-access-4qfw4\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.656833 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.656821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.656980 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:57.656902 2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 17:13:57.656980 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.656937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.656980 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:57.656957 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-tls podName:10e0eb75-b3c8-428d-92d6-f27dc9829f89 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:58.156943028 +0000 UTC m=+315.346716390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-r42mm" (UID: "10e0eb75-b3c8-428d-92d6-f27dc9829f89") : secret "openshift-state-metrics-tls" not found Jul 28 17:13:57.657125 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.656985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.657125 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-tls\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657125 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10e0eb75-b3c8-428d-92d6-f27dc9829f89-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.657279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-wtmp\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7nh\" (UniqueName: \"kubernetes.io/projected/10e0eb75-b3c8-428d-92d6-f27dc9829f89-kube-api-access-qq7nh\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.657279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.657279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657256 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-sys\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-root\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657343 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-wtmp\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-textfile\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:57.657412 2572 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84ecc9cb-5524-46d0-b51c-31be4077f6f9-metrics-client-ca\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657504 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:13:57.657476 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-tls podName:6bc18ab9-5065-4e75-b889-fdbb998f1aeb nodeName:}" failed. No retries permitted until 2026-07-28 17:13:58.157451938 +0000 UTC m=+315.347225318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-zg7pm" (UID: "6bc18ab9-5065-4e75-b889-fdbb998f1aeb") : secret "kube-state-metrics-tls" not found Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86k4\" (UniqueName: \"kubernetes.io/projected/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-api-access-d86k4\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657597 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-textfile\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.657884 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657843 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10e0eb75-b3c8-428d-92d6-f27dc9829f89-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.658292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.657958 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84ecc9cb-5524-46d0-b51c-31be4077f6f9-metrics-client-ca\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.658292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.658028 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-sys\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.658292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.658131 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.658292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.658192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84ecc9cb-5524-46d0-b51c-31be4077f6f9-root\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.658523 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.658369 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.658875 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.658848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.659908 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.659881 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-tls\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.659990 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.659968 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.660030 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.660002 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84ecc9cb-5524-46d0-b51c-31be4077f6f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.660067 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.660034 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.670232 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.670206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfw4\" (UniqueName: \"kubernetes.io/projected/84ecc9cb-5524-46d0-b51c-31be4077f6f9-kube-api-access-4qfw4\") pod \"node-exporter-4jhlt\" (UID: \"84ecc9cb-5524-46d0-b51c-31be4077f6f9\") " pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.671621 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.671601 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7nh\" (UniqueName: \"kubernetes.io/projected/10e0eb75-b3c8-428d-92d6-f27dc9829f89-kube-api-access-qq7nh\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:57.671774 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.671756 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86k4\" (UniqueName: \"kubernetes.io/projected/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-api-access-d86k4\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:57.743120 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:57.743100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4jhlt" Jul 28 17:13:57.755128 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:57.755099 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ecc9cb_5524_46d0_b51c_31be4077f6f9.slice/crio-450b1bffd2298503b829c4ab5088125752a5f1069d69adea53a2566ffb04d289 WatchSource:0}: Error finding container 450b1bffd2298503b829c4ab5088125752a5f1069d69adea53a2566ffb04d289: Status 404 returned error can't find the container with id 450b1bffd2298503b829c4ab5088125752a5f1069d69adea53a2566ffb04d289 Jul 28 17:13:58.161029 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.161001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:58.161190 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.161073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:58.163370 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.163349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10e0eb75-b3c8-428d-92d6-f27dc9829f89-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-r42mm\" (UID: \"10e0eb75-b3c8-428d-92d6-f27dc9829f89\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:58.163475 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.163429 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bc18ab9-5065-4e75-b889-fdbb998f1aeb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zg7pm\" (UID: \"6bc18ab9-5065-4e75-b889-fdbb998f1aeb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:58.303967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.303938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" Jul 28 17:13:58.315677 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.315652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" Jul 28 17:13:58.394740 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.394575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jhlt" event={"ID":"84ecc9cb-5524-46d0-b51c-31be4077f6f9","Type":"ContainerStarted","Data":"450b1bffd2298503b829c4ab5088125752a5f1069d69adea53a2566ffb04d289"} Jul 28 17:13:58.476654 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.476292 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm"] Jul 28 17:13:58.703722 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:58.703695 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm"] Jul 28 17:13:58.707995 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:13:58.707949 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc18ab9_5065_4e75_b889_fdbb998f1aeb.slice/crio-a56727bc2b7d9cd83b9f07748d92952a1da0b8e27f92ede03806407f6ebea990 WatchSource:0}: Error finding container a56727bc2b7d9cd83b9f07748d92952a1da0b8e27f92ede03806407f6ebea990: Status 404 returned error can't find the container with id a56727bc2b7d9cd83b9f07748d92952a1da0b8e27f92ede03806407f6ebea990 Jul 28 17:13:59.401520 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:59.401480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" event={"ID":"10e0eb75-b3c8-428d-92d6-f27dc9829f89","Type":"ContainerStarted","Data":"40fb281e28e331ad5ad23cb9eaebab60f9b20ed0a8700d191f50d0a7c1ceb956"} Jul 28 17:13:59.401969 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:59.401554 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" event={"ID":"10e0eb75-b3c8-428d-92d6-f27dc9829f89","Type":"ContainerStarted","Data":"4efeee46822014699cc55b404b9c6db331ae5395e9536ae42e04812c560fc30a"} Jul 28 17:13:59.401969 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:59.401584 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" event={"ID":"10e0eb75-b3c8-428d-92d6-f27dc9829f89","Type":"ContainerStarted","Data":"6c17b8bedf9afe2be6bff259a2d38f295de4f5d0deed9ce00b951263db38105b"} Jul 28 17:13:59.403270 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:59.403190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" event={"ID":"6bc18ab9-5065-4e75-b889-fdbb998f1aeb","Type":"ContainerStarted","Data":"a56727bc2b7d9cd83b9f07748d92952a1da0b8e27f92ede03806407f6ebea990"} Jul 28 17:13:59.405338 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:59.405311 2572 generic.go:356] "Generic (PLEG): container finished" podID="84ecc9cb-5524-46d0-b51c-31be4077f6f9" containerID="0eace626b23a382821fb8b0cdb4ea797d838d45f49c1e14340d1724fec98dd95" exitCode=0 Jul 28 17:13:59.405460 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:13:59.405358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jhlt" event={"ID":"84ecc9cb-5524-46d0-b51c-31be4077f6f9","Type":"ContainerDied","Data":"0eace626b23a382821fb8b0cdb4ea797d838d45f49c1e14340d1724fec98dd95"} Jul 28 17:14:00.412221 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.412128 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jhlt" event={"ID":"84ecc9cb-5524-46d0-b51c-31be4077f6f9","Type":"ContainerStarted","Data":"ad75c80908081ae093494f9b27396776f2320a6e99dc445ddde6bac2114c3d97"} Jul 28 17:14:00.412221 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.412201 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4jhlt" event={"ID":"84ecc9cb-5524-46d0-b51c-31be4077f6f9","Type":"ContainerStarted","Data":"1c7273157dc9a1424b3ff7f8a790d64fd071fd1a86788570cd7ad7cef82fbd99"} Jul 28 17:14:00.414453 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.414422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" event={"ID":"10e0eb75-b3c8-428d-92d6-f27dc9829f89","Type":"ContainerStarted","Data":"26628210ea585491ba0c12be121cb0e9dbb59f09b284ce3b8912bb0addf5e2c4"} Jul 28 17:14:00.416929 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.416904 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" event={"ID":"6bc18ab9-5065-4e75-b889-fdbb998f1aeb","Type":"ContainerStarted","Data":"39bb66c3c71119c42dddee728db8ba7747d0732cff13bc02641c04d27b5aecc6"} Jul 28 17:14:00.417042 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.416945 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" event={"ID":"6bc18ab9-5065-4e75-b889-fdbb998f1aeb","Type":"ContainerStarted","Data":"bcaf8f3aa366b0be4ef557a3ab47c6047e524aaf6788ca97de64aff1476ee3fb"} Jul 28 17:14:00.417042 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.416963 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" event={"ID":"6bc18ab9-5065-4e75-b889-fdbb998f1aeb","Type":"ContainerStarted","Data":"3896f992bd6e1636c7de891806f13ac8fdfbc7f069c68e18639f99c2da145ba3"} Jul 28 17:14:00.442668 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.442626 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4jhlt" podStartSLOduration=2.390102456 podStartE2EDuration="3.442612407s" totalImagesPullingTime="986.292534ms" totalInitContainerRuntime="66.217417ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:00.441542686 +0000 UTC m=+317.631316072" watchObservedRunningTime="2026-07-28 17:14:00.442612407 +0000 UTC m=+317.632385792" Jul 28 17:14:00.460615 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.460573 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zg7pm" podStartSLOduration=2.042355534 podStartE2EDuration="3.460557961s" totalImagesPullingTime="1.418202427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:00.459576662 +0000 UTC m=+317.649350050" watchObservedRunningTime="2026-07-28 17:14:00.460557961 +0000 UTC m=+317.650331345" Jul 28 17:14:00.480210 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:00.480002 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-r42mm" podStartSLOduration=2.17277271 podStartE2EDuration="3.479985627s" totalImagesPullingTime="1.307212917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:00.479573975 +0000 UTC m=+317.669347360" watchObservedRunningTime="2026-07-28 17:14:00.479985627 +0000 UTC m=+317.669759014" Jul 28 17:14:01.919453 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.919382 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-769795f47b-ltlpf"] Jul 28 17:14:01.922718 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.922699 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:01.926585 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.926565 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 17:14:01.927261 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.927235 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4dkd176rec3ts\"" Jul 28 17:14:01.927261 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.927248 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 17:14:01.927429 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.927244 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 17:14:01.927429 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.927281 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 17:14:01.927429 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.927353 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-btdnt\"" Jul 28 17:14:01.937564 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:01.937523 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-769795f47b-ltlpf"] Jul 28 17:14:02.097630 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097591 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx6jj\" (UniqueName: \"kubernetes.io/projected/f555f995-2e0f-40ba-b0da-a78275b4a87e-kube-api-access-cx6jj\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.097793 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-secret-metrics-server-client-certs\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.097793 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f555f995-2e0f-40ba-b0da-a78275b4a87e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.097793 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-secret-metrics-server-tls\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.097793 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f555f995-2e0f-40ba-b0da-a78275b4a87e-metrics-server-audit-profiles\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.097954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097814 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f555f995-2e0f-40ba-b0da-a78275b4a87e-audit-log\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.097954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.097857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-client-ca-bundle\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.121818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.121781 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp"] Jul 28 17:14:02.125123 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.125108 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.127151 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.127132 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mbkzr\"" Jul 28 17:14:02.127465 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.127450 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 17:14:02.168095 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.168065 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp"] Jul 28 17:14:02.198902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.198817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f555f995-2e0f-40ba-b0da-a78275b4a87e-audit-log\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.198902 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.198851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-client-ca-bundle\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199106 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.198906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx6jj\" (UniqueName: \"kubernetes.io/projected/f555f995-2e0f-40ba-b0da-a78275b4a87e-kube-api-access-cx6jj\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199106 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.198961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-secret-metrics-server-client-certs\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199106 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.199002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f555f995-2e0f-40ba-b0da-a78275b4a87e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199106 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.199046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-secret-metrics-server-tls\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199106 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.199093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f555f995-2e0f-40ba-b0da-a78275b4a87e-metrics-server-audit-profiles\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199328 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.199303 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f555f995-2e0f-40ba-b0da-a78275b4a87e-audit-log\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199771 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.199742 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f555f995-2e0f-40ba-b0da-a78275b4a87e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.199984 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.199965 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f555f995-2e0f-40ba-b0da-a78275b4a87e-metrics-server-audit-profiles\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.201459 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.201441 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-client-ca-bundle\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.201580 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.201559 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-secret-metrics-server-client-certs\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.201619 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.201561 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f555f995-2e0f-40ba-b0da-a78275b4a87e-secret-metrics-server-tls\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.208817 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.208793 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx6jj\" (UniqueName: \"kubernetes.io/projected/f555f995-2e0f-40ba-b0da-a78275b4a87e-kube-api-access-cx6jj\") pod \"metrics-server-769795f47b-ltlpf\" (UID: \"f555f995-2e0f-40ba-b0da-a78275b4a87e\") " pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.231510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.231487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:02.299873 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.299839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr56w\" (UniqueName: \"kubernetes.io/projected/da27684e-b333-4b84-9672-0c29deb8ae54-kube-api-access-dr56w\") pod \"monitoring-plugin-68b7fc9574-jcxsp\" (UID: \"da27684e-b333-4b84-9672-0c29deb8ae54\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.300084 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.300060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da27684e-b333-4b84-9672-0c29deb8ae54-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jcxsp\" (UID: \"da27684e-b333-4b84-9672-0c29deb8ae54\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.366018 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.365984 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-769795f47b-ltlpf"] Jul 28 17:14:02.368955 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:14:02.368929 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf555f995_2e0f_40ba_b0da_a78275b4a87e.slice/crio-6e0eefed92b284aa5496873cee63ed1a08ccedf54d85bff46cbd7f6fc03b1d3e WatchSource:0}: Error finding container 6e0eefed92b284aa5496873cee63ed1a08ccedf54d85bff46cbd7f6fc03b1d3e: Status 404 returned error can't find the container with id 6e0eefed92b284aa5496873cee63ed1a08ccedf54d85bff46cbd7f6fc03b1d3e Jul 28 17:14:02.400921 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.400896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da27684e-b333-4b84-9672-0c29deb8ae54-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jcxsp\" (UID: \"da27684e-b333-4b84-9672-0c29deb8ae54\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.401027 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.400943 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr56w\" (UniqueName: \"kubernetes.io/projected/da27684e-b333-4b84-9672-0c29deb8ae54-kube-api-access-dr56w\") pod \"monitoring-plugin-68b7fc9574-jcxsp\" (UID: \"da27684e-b333-4b84-9672-0c29deb8ae54\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.403162 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.403135 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da27684e-b333-4b84-9672-0c29deb8ae54-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jcxsp\" (UID: \"da27684e-b333-4b84-9672-0c29deb8ae54\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.411246 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.411223 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr56w\" (UniqueName: \"kubernetes.io/projected/da27684e-b333-4b84-9672-0c29deb8ae54-kube-api-access-dr56w\") pod \"monitoring-plugin-68b7fc9574-jcxsp\" (UID: \"da27684e-b333-4b84-9672-0c29deb8ae54\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.425908 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.425885 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" event={"ID":"f555f995-2e0f-40ba-b0da-a78275b4a87e","Type":"ContainerStarted","Data":"6e0eefed92b284aa5496873cee63ed1a08ccedf54d85bff46cbd7f6fc03b1d3e"} Jul 28 17:14:02.434232 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.434205 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:02.561591 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:02.561569 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp"] Jul 28 17:14:02.563561 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:14:02.563529 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda27684e_b333_4b84_9672_0c29deb8ae54.slice/crio-a52c84244512d3d63895c79242a1f8bc348c51393e43be67b9883a4a20e5d51a WatchSource:0}: Error finding container a52c84244512d3d63895c79242a1f8bc348c51393e43be67b9883a4a20e5d51a: Status 404 returned error can't find the container with id a52c84244512d3d63895c79242a1f8bc348c51393e43be67b9883a4a20e5d51a Jul 28 17:14:03.432678 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.432635 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" event={"ID":"da27684e-b333-4b84-9672-0c29deb8ae54","Type":"ContainerStarted","Data":"a52c84244512d3d63895c79242a1f8bc348c51393e43be67b9883a4a20e5d51a"} Jul 28 17:14:03.817664 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.816114 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:14:03.821091 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.821068 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.826734 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.826707 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:14:03.828921 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.826933 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:14:03.828921 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.827056 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:14:03.828921 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.827290 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:14:03.829466 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.829229 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c282c4kfpdeev\"" Jul 28 17:14:03.829466 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.829291 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:14:03.829567 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.829480 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wm4hv\"" Jul 28 17:14:03.829626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.829604 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:14:03.829719 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.829700 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:14:03.829924 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.829893 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:14:03.830130 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.830110 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:14:03.830315 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.830279 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:14:03.830416 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.830368 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:14:03.830842 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.830698 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:14:03.831911 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.831863 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:14:03.834504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.834479 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:14:03.839839 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.839813 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:14:03.919287 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9v5z\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-kube-api-access-m9v5z\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919703 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-config\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919844 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.919918 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919914 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.920123 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.919938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:03.920123 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:03.920011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022153 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-config\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022438 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022612 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022681 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022837 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022753 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022837 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022933 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9v5z\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-kube-api-access-m9v5z\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022933 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.022933 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023072 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023072 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.022967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023072 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023028 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023072 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023024 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023072 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023053 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023297 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023297 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023297 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023168 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023297 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.023297 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.023232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.024232 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.024202 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.025049 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.024889 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.027954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.026025 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-config\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.027954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.026475 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.027954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.026551 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.027954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.027011 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.028420 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.028362 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.029135 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.029003 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.030384 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.030318 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.030930 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.030900 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.031503 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.031479 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.031690 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.031669 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.033154 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.033063 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.033331 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.033312 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.033698 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.033678 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.033748 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.033686 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.034748 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.034701 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.035290 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.035266 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.035981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.035955 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9v5z\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-kube-api-access-m9v5z\") pod \"prometheus-k8s-0\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.133818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.133487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:04.437685 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.437643 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" event={"ID":"da27684e-b333-4b84-9672-0c29deb8ae54","Type":"ContainerStarted","Data":"67fc2a9089fa914bf490b2e2630022597d83d8442f09e8ce84b19f9610968f9a"} Jul 28 17:14:04.438055 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.437925 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:04.439996 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.439966 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" event={"ID":"f555f995-2e0f-40ba-b0da-a78275b4a87e","Type":"ContainerStarted","Data":"bc3a0ed7589b4822b50a6bdd0cc59db79b6b44fa9a610aadc8db1486ace5cbf7"} Jul 28 17:14:04.443737 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.443706 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" Jul 28 17:14:04.459420 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.458498 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jcxsp" podStartSLOduration=0.695961088 podStartE2EDuration="2.458485498s" totalImagesPullingTime="1.76252441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:04.456915752 +0000 UTC m=+321.646689137" watchObservedRunningTime="2026-07-28 17:14:04.458485498 +0000 UTC m=+321.648258930" Jul 28 17:14:04.470970 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.470882 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:14:04.473054 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:14:04.473029 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3457fa_3f26_4551_aa80_ff5772c035c4.slice/crio-71c4368c284f8443acd67d2423d6253f469b602269cf38198034987324631699 WatchSource:0}: Error finding container 71c4368c284f8443acd67d2423d6253f469b602269cf38198034987324631699: Status 404 returned error can't find the container with id 71c4368c284f8443acd67d2423d6253f469b602269cf38198034987324631699 Jul 28 17:14:04.487359 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:04.487320 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" podStartSLOduration=1.5330887629999999 podStartE2EDuration="3.487307345s" totalImagesPullingTime="1.954218582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:04.486138835 +0000 UTC m=+321.675912233" watchObservedRunningTime="2026-07-28 17:14:04.487307345 +0000 UTC m=+321.677080730" Jul 28 17:14:05.446200 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:05.446160 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"71c4368c284f8443acd67d2423d6253f469b602269cf38198034987324631699"} Jul 28 17:14:05.625716 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:05.625686 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d68454b6-wdf5b"] Jul 28 17:14:05.626003 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:14:05.625983 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" podUID="bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" Jul 28 17:14:06.451200 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.451167 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" exitCode=0 Jul 28 17:14:06.451650 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.451233 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:14:06.451650 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.451255 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} Jul 28 17:14:06.457819 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.457802 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:14:06.546448 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546420 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-image-registry-private-configuration\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546480 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-installation-pull-secrets\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546501 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-certificates\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546517 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhnj7\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-kube-api-access-dhnj7\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546549 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-ca-trust-extracted\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546573 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-bound-sa-token\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546739 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546611 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-trusted-ca\") pod \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\" (UID: \"bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab\") " Jul 28 17:14:06.546905 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546878 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:14:06.546966 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.546934 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:14:06.547332 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.547305 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:14:06.547790 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.547764 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-certificates\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:06.547882 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.547795 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-ca-trust-extracted\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:06.547882 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.547809 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-trusted-ca\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:06.549155 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.549128 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:14:06.549242 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.549227 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:14:06.549321 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.549298 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-kube-api-access-dhnj7" (OuterVolumeSpecName: "kube-api-access-dhnj7") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "kube-api-access-dhnj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:14:06.549652 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.549630 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" (UID: "bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:14:06.648965 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.648943 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-installation-pull-secrets\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:06.648965 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.648962 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhnj7\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-kube-api-access-dhnj7\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:06.649081 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.648972 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-bound-sa-token\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:06.649081 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:06.648982 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-image-registry-private-configuration\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:07.455406 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:07.455357 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d68454b6-wdf5b" Jul 28 17:14:07.493865 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:07.493831 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d68454b6-wdf5b"] Jul 28 17:14:07.501609 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:07.501578 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65d68454b6-wdf5b"] Jul 28 17:14:07.556901 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:07.556876 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab-registry-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:14:09.337474 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:09.337437 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab" path="/var/lib/kubelet/pods/bcfdf9ba-a2f1-4f5d-91a8-e13c6361f0ab/volumes" Jul 28 17:14:10.470835 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:10.470804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} Jul 28 17:14:10.471304 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:10.470874 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} Jul 28 17:14:12.481548 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:12.481510 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} Jul 28 17:14:12.482019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:12.481590 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} Jul 28 17:14:12.482019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:12.481636 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} Jul 28 17:14:12.482019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:12.481682 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerStarted","Data":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} Jul 28 17:14:12.520254 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:12.520196 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.436943226 podStartE2EDuration="9.520180951s" totalImagesPullingTime="6.06330885s" totalInitContainerRuntime="19.928875ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:12.5176873 +0000 UTC m=+329.707460719" watchObservedRunningTime="2026-07-28 17:14:12.520180951 +0000 UTC m=+329.709954336" Jul 28 17:14:14.133941 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:14.133909 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:14:22.232339 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:22.232302 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:22.232715 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:22.232362 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:22.528950 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:22.528893 2572 generic.go:356] "Generic (PLEG): container finished" podID="9cf606e8-a632-48cc-a9b1-01bd9200de94" containerID="6f440cac3ef8f6f5ca4fbe1ccc8dceeeb733f3736932556c19bb8e68bbcb0e4f" exitCode=0 Jul 28 17:14:22.529060 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:22.528970 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" event={"ID":"9cf606e8-a632-48cc-a9b1-01bd9200de94","Type":"ContainerDied","Data":"6f440cac3ef8f6f5ca4fbe1ccc8dceeeb733f3736932556c19bb8e68bbcb0e4f"} Jul 28 17:14:22.529309 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:22.529293 2572 scope.go:117] "RemoveContainer" containerID="6f440cac3ef8f6f5ca4fbe1ccc8dceeeb733f3736932556c19bb8e68bbcb0e4f" Jul 28 17:14:23.533772 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:23.533739 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kgzlw" event={"ID":"9cf606e8-a632-48cc-a9b1-01bd9200de94","Type":"ContainerStarted","Data":"87c6d9ac55d5ed34f50d852bc650b3bf3025be598a87104b94fee4fa8f62875c"} Jul 28 17:14:25.344432 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:25.344403 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-bp2dz_bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8/cluster-monitoring-operator/0.log" Jul 28 17:14:25.534888 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:25.534859 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zg7pm_6bc18ab9-5065-4e75-b889-fdbb998f1aeb/kube-state-metrics/0.log" Jul 28 17:14:25.734295 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:25.734230 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zg7pm_6bc18ab9-5065-4e75-b889-fdbb998f1aeb/kube-rbac-proxy-main/0.log" Jul 28 17:14:25.934609 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:25.934583 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zg7pm_6bc18ab9-5065-4e75-b889-fdbb998f1aeb/kube-rbac-proxy-self/0.log" Jul 28 17:14:26.133961 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:26.133938 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-769795f47b-ltlpf_f555f995-2e0f-40ba-b0da-a78275b4a87e/metrics-server/0.log" Jul 28 17:14:26.334494 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:26.334463 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-jcxsp_da27684e-b333-4b84-9672-0c29deb8ae54/monitoring-plugin/0.log" Jul 28 17:14:27.133533 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:27.133506 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jhlt_84ecc9cb-5524-46d0-b51c-31be4077f6f9/init-textfile/0.log" Jul 28 17:14:27.338422 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:27.338400 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jhlt_84ecc9cb-5524-46d0-b51c-31be4077f6f9/node-exporter/0.log" Jul 28 17:14:27.536774 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:27.536754 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jhlt_84ecc9cb-5524-46d0-b51c-31be4077f6f9/kube-rbac-proxy/0.log" Jul 28 17:14:28.335818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:28.335788 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-r42mm_10e0eb75-b3c8-428d-92d6-f27dc9829f89/kube-rbac-proxy-main/0.log" Jul 28 17:14:28.535007 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:28.534982 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-r42mm_10e0eb75-b3c8-428d-92d6-f27dc9829f89/kube-rbac-proxy-self/0.log" Jul 28 17:14:28.740885 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:28.740821 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-r42mm_10e0eb75-b3c8-428d-92d6-f27dc9829f89/openshift-state-metrics/0.log" Jul 28 17:14:28.935953 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:28.935925 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/init-config-reloader/0.log" Jul 28 17:14:29.135764 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:29.135739 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/prometheus/0.log" Jul 28 17:14:29.339690 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:29.339663 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/config-reloader/0.log" Jul 28 17:14:29.538802 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:29.538781 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/thanos-sidecar/0.log" Jul 28 17:14:29.734858 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:29.734832 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/kube-rbac-proxy-web/0.log" Jul 28 17:14:29.936937 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:29.936863 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/kube-rbac-proxy/0.log" Jul 28 17:14:30.147522 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:30.147499 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c3457fa-3f26-4551-aa80-ff5772c035c4/kube-rbac-proxy-thanos/0.log" Jul 28 17:14:30.337949 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:30.337910 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-b8l29_3bd52883-679e-45e7-91d6-164cc3d9eb21/prometheus-operator/0.log" Jul 28 17:14:30.535061 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:30.535033 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-b8l29_3bd52883-679e-45e7-91d6-164cc3d9eb21/kube-rbac-proxy/0.log" Jul 28 17:14:35.134895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:35.134868 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npp2t_aca3ad49-96eb-4f52-b11e-4bea99d26f5d/dns-node-resolver/0.log" Jul 28 17:14:37.591941 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:37.591906 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1bb1ced-d125-4861-b579-954c9e7558ba" containerID="4cc4d39e515a21dac5d996b9273cd23d6bf8b59d81335a6c0461b2fc37375aa0" exitCode=0 Jul 28 17:14:37.592335 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:37.591978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" event={"ID":"a1bb1ced-d125-4861-b579-954c9e7558ba","Type":"ContainerDied","Data":"4cc4d39e515a21dac5d996b9273cd23d6bf8b59d81335a6c0461b2fc37375aa0"} Jul 28 17:14:37.592383 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:37.592363 2572 scope.go:117] "RemoveContainer" containerID="4cc4d39e515a21dac5d996b9273cd23d6bf8b59d81335a6c0461b2fc37375aa0" Jul 28 17:14:37.593773 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:37.593711 2572 generic.go:356] "Generic (PLEG): container finished" podID="0578c2fb-d319-4d60-a4b7-465db4a7a3e4" containerID="4328c95c4473078177b60ffc7f7755e69e454ec4baffe0267d25318a0e952a28" exitCode=0 Jul 28 17:14:37.593773 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:37.593746 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" event={"ID":"0578c2fb-d319-4d60-a4b7-465db4a7a3e4","Type":"ContainerDied","Data":"4328c95c4473078177b60ffc7f7755e69e454ec4baffe0267d25318a0e952a28"} Jul 28 17:14:37.594064 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:37.594022 2572 scope.go:117] "RemoveContainer" containerID="4328c95c4473078177b60ffc7f7755e69e454ec4baffe0267d25318a0e952a28" Jul 28 17:14:38.600468 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:38.600426 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m" event={"ID":"a1bb1ced-d125-4861-b579-954c9e7558ba","Type":"ContainerStarted","Data":"2cf20172d76b336c33ba5b102c181d024dce3ade3445156c899f036d8381e40f"} Jul 28 17:14:38.602422 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:38.602368 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-km5nn" event={"ID":"0578c2fb-d319-4d60-a4b7-465db4a7a3e4","Type":"ContainerStarted","Data":"984c17b76a2723885b36fbe8fb71ebfadb4d23d636456c19fc23a32e8bdd0f3c"} Jul 28 17:14:42.237817 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:42.237786 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:42.241716 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:42.241695 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-769795f47b-ltlpf" Jul 28 17:14:56.168549 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:56.168512 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:14:56.170835 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:56.170815 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61deeab5-0b0e-4a34-80ea-e476ec574ee7-metrics-certs\") pod \"network-metrics-daemon-m4kwh\" (UID: \"61deeab5-0b0e-4a34-80ea-e476ec574ee7\") " pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:14:56.434986 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:56.434910 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-24g9l\"" Jul 28 17:14:56.443474 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:56.443456 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4kwh" Jul 28 17:14:56.565914 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:56.565891 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4kwh"] Jul 28 17:14:56.568052 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:14:56.568022 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61deeab5_0b0e_4a34_80ea_e476ec574ee7.slice/crio-9bac10be96f51e93dfb47f7e6fcc2d2f1153f395c32efb8b29eaac54039bae42 WatchSource:0}: Error finding container 9bac10be96f51e93dfb47f7e6fcc2d2f1153f395c32efb8b29eaac54039bae42: Status 404 returned error can't find the container with id 9bac10be96f51e93dfb47f7e6fcc2d2f1153f395c32efb8b29eaac54039bae42 Jul 28 17:14:56.674358 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:56.674328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4kwh" event={"ID":"61deeab5-0b0e-4a34-80ea-e476ec574ee7","Type":"ContainerStarted","Data":"9bac10be96f51e93dfb47f7e6fcc2d2f1153f395c32efb8b29eaac54039bae42"} Jul 28 17:14:57.680137 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:57.680100 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4kwh" event={"ID":"61deeab5-0b0e-4a34-80ea-e476ec574ee7","Type":"ContainerStarted","Data":"7926efc56d3182bf51969523b1c912769374b305b5d3419491c27de22cd002ad"} Jul 28 17:14:57.680502 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:57.680156 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4kwh" event={"ID":"61deeab5-0b0e-4a34-80ea-e476ec574ee7","Type":"ContainerStarted","Data":"599ae5574dc6a6c9768bb87f6a0c7b8a387127c488b3da24d480588b06fd4ea3"} Jul 28 17:14:57.696280 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:14:57.696201 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m4kwh" podStartSLOduration=373.823341311 podStartE2EDuration="6m14.696185381s" totalImagesPullingTime="872.84407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:57.695578512 +0000 UTC m=+374.885351898" watchObservedRunningTime="2026-07-28 17:14:57.696185381 +0000 UTC m=+374.885958766" Jul 28 17:15:04.133657 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:04.133612 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:04.149622 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:04.149598 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:04.720561 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:04.720534 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.244670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.244601 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:15:22.245490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.245026 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="prometheus" containerID="cri-o://983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" gracePeriod=600 Jul 28 17:15:22.245490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.245110 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-web" containerID="cri-o://0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" gracePeriod=600 Jul 28 17:15:22.245490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.245124 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy" containerID="cri-o://2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" gracePeriod=600 Jul 28 17:15:22.245490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.245176 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="config-reloader" containerID="cri-o://8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" gracePeriod=600 Jul 28 17:15:22.245490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.245136 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="thanos-sidecar" containerID="cri-o://a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" gracePeriod=600 Jul 28 17:15:22.245490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.245237 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" gracePeriod=600 Jul 28 17:15:22.502518 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.502461 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592028 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-db\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592080 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-trusted-ca-bundle\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592118 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592154 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-thanos-prometheus-http-client-file\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592181 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-grpc-tls\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592211 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-kubelet-serving-ca-bundle\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592249 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592275 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-config\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592296 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-config-out\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592322 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-metrics-client-certs\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.592478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592355 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-0\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592517 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-2\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592556 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-1\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592587 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-tls-assets\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592619 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-web-config\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592667 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9v5z\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-kube-api-access-m9v5z\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592693 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-kube-rbac-proxy\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592756 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-serving-certs-ca-bundle\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592786 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-metrics-client-ca\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592818 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-tls\") pod \"7c3457fa-3f26-4551-aa80-ff5772c035c4\" (UID: \"7c3457fa-3f26-4551-aa80-ff5772c035c4\") " Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592850 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592892 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.593100 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.592915 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.593715 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.593110 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.593715 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.593151 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.593715 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.593169 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.593715 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.593185 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.595633 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.595577 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.595938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.595856 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.596042 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.595972 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.596042 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.596015 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.596165 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.596125 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:15:22.597289 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.597244 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-config" (OuterVolumeSpecName: "config") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.597289 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.597247 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:15:22.597812 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.597783 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:22.598235 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.598210 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-config-out" (OuterVolumeSpecName: "config-out") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:22.598541 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.598502 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.598541 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.598524 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.598685 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.598545 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.598911 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.598892 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-kube-api-access-m9v5z" (OuterVolumeSpecName: "kube-api-access-m9v5z") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "kube-api-access-m9v5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:15:22.598974 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.598893 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.599135 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.599117 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.610214 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.610192 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-web-config" (OuterVolumeSpecName: "web-config") pod "7c3457fa-3f26-4551-aa80-ff5772c035c4" (UID: "7c3457fa-3f26-4551-aa80-ff5772c035c4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:15:22.693869 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693838 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-db\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.693869 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693864 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.693869 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693874 2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693884 2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-grpc-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693894 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693902 2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-config\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693910 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3457fa-3f26-4551-aa80-ff5772c035c4-config-out\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693918 2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-metrics-client-certs\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693927 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693935 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693942 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-tls-assets\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693951 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-web-config\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693959 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9v5z\" (UniqueName: \"kubernetes.io/projected/7c3457fa-3f26-4551-aa80-ff5772c035c4-kube-api-access-m9v5z\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693967 2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-kube-rbac-proxy\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693975 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693984 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c3457fa-3f26-4551-aa80-ff5772c035c4-configmap-metrics-client-ca\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.694052 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.693992 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c3457fa-3f26-4551-aa80-ff5772c035c4-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:15:22.782131 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782057 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" exitCode=0 Jul 28 17:15:22.782131 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782079 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" exitCode=0 Jul 28 17:15:22.782131 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782086 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" exitCode=0 Jul 28 17:15:22.782131 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782091 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" exitCode=0 Jul 28 17:15:22.782131 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782097 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" exitCode=0 Jul 28 17:15:22.782131 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782103 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" exitCode=0 Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782135 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782143 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782231 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782310 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782320 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782423 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} Jul 28 17:15:22.782530 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.782466 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c3457fa-3f26-4551-aa80-ff5772c035c4","Type":"ContainerDied","Data":"71c4368c284f8443acd67d2423d6253f469b602269cf38198034987324631699"} Jul 28 17:15:22.791381 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.791306 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.798243 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.798229 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.804642 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.804626 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.810606 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.810586 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:15:22.811384 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.811374 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.816910 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.816891 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:15:22.818378 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.818360 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.825233 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.825208 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.831605 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.831591 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.831838 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.831820 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.831881 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.831846 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} err="failed to get container status \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" Jul 28 17:15:22.831881 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.831875 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.832081 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.832067 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.832128 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832084 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} err="failed to get container status \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" Jul 28 17:15:22.832128 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832104 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.832340 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.832323 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.832425 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832357 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} err="failed to get container status \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" Jul 28 17:15:22.832425 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832380 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.832625 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.832610 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.832660 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832628 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} err="failed to get container status \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" Jul 28 17:15:22.832660 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832644 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.832828 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.832809 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.832869 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832834 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} err="failed to get container status \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" Jul 28 17:15:22.832869 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.832847 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.833062 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.833048 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.833115 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833065 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} err="failed to get container status \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" Jul 28 17:15:22.833115 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833076 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.833312 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:22.833297 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.833417 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833313 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} err="failed to get container status \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" Jul 28 17:15:22.833417 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833324 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.833551 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833529 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} err="failed to get container status \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" Jul 28 17:15:22.833588 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833553 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.833754 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833734 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} err="failed to get container status \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" Jul 28 17:15:22.833754 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833752 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.833936 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833920 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} err="failed to get container status \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" Jul 28 17:15:22.833983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.833937 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.834116 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834097 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} err="failed to get container status \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" Jul 28 17:15:22.834176 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834118 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.834349 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834332 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} err="failed to get container status \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" Jul 28 17:15:22.834442 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834352 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.834571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834555 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} err="failed to get container status \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" Jul 28 17:15:22.834613 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834572 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.834752 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834727 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} err="failed to get container status \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" Jul 28 17:15:22.834796 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834753 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.834936 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834919 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} err="failed to get container status \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" Jul 28 17:15:22.834978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.834938 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.835127 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835112 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} err="failed to get container status \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" Jul 28 17:15:22.835165 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835127 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.835319 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835296 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} err="failed to get container status \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" Jul 28 17:15:22.835356 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835321 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.835571 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835540 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} err="failed to get container status \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" Jul 28 17:15:22.835626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835571 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.835776 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835754 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} err="failed to get container status \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" Jul 28 17:15:22.835845 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835779 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.835993 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835978 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} err="failed to get container status \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" Jul 28 17:15:22.835993 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.835993 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.836206 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836192 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} err="failed to get container status \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" Jul 28 17:15:22.836206 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836205 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.836360 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836345 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} err="failed to get container status \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" Jul 28 17:15:22.836360 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836359 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.836583 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836565 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} err="failed to get container status \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" Jul 28 17:15:22.836655 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836584 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.836807 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836789 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} err="failed to get container status \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" Jul 28 17:15:22.836851 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836808 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.836997 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836978 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} err="failed to get container status \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" Jul 28 17:15:22.837063 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.836998 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.837187 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837170 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} err="failed to get container status \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" Jul 28 17:15:22.837246 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837189 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.837363 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837347 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} err="failed to get container status \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" Jul 28 17:15:22.837423 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837364 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.837570 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837553 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} err="failed to get container status \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" Jul 28 17:15:22.837638 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837572 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.837802 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837783 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} err="failed to get container status \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" Jul 28 17:15:22.837858 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837803 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.837992 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837972 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} err="failed to get container status \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" Jul 28 17:15:22.838035 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.837993 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.838156 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838141 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} err="failed to get container status \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" Jul 28 17:15:22.838203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838156 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.838360 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838340 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} err="failed to get container status \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" Jul 28 17:15:22.838360 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838359 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.838540 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838524 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} err="failed to get container status \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" Jul 28 17:15:22.838589 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838542 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.838741 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838723 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} err="failed to get container status \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" Jul 28 17:15:22.838794 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838741 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.838937 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838919 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} err="failed to get container status \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" Jul 28 17:15:22.838937 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.838937 2572 scope.go:117] "RemoveContainer" containerID="6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f" Jul 28 17:15:22.839130 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839110 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f"} err="failed to get container status \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": rpc error: code = NotFound desc = could not find container \"6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f\": container with ID starting with 6b1284685c9b72d8872823a9c624bc6505a5fd0d4e5b4286b9d87dca84641f6f not found: ID does not exist" Jul 28 17:15:22.839184 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839132 2572 scope.go:117] "RemoveContainer" containerID="2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a" Jul 28 17:15:22.839342 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839326 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a"} err="failed to get container status \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": rpc error: code = NotFound desc = could not find container \"2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a\": container with ID starting with 2da1a121e369ff062525ce00ecc811796124e8f13be1692a086bd5a050d1bd4a not found: ID does not exist" Jul 28 17:15:22.839383 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839342 2572 scope.go:117] "RemoveContainer" containerID="0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f" Jul 28 17:15:22.839518 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839502 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f"} err="failed to get container status \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": rpc error: code = NotFound desc = could not find container \"0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f\": container with ID starting with 0854063d8f54a45b5cfc2f72d7b1f86d06061992ff9fd078ec1e36a933e9071f not found: ID does not exist" Jul 28 17:15:22.839562 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839518 2572 scope.go:117] "RemoveContainer" containerID="a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59" Jul 28 17:15:22.839685 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839666 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59"} err="failed to get container status \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": rpc error: code = NotFound desc = could not find container \"a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59\": container with ID starting with a527acd9b0a8533315021758907d59e8403d5689ee0e0b0143cb72d169f8de59 not found: ID does not exist" Jul 28 17:15:22.839725 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839686 2572 scope.go:117] "RemoveContainer" containerID="8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8" Jul 28 17:15:22.839878 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839863 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8"} err="failed to get container status \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": rpc error: code = NotFound desc = could not find container \"8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8\": container with ID starting with 8d520a8b1daddf220afabed4f12f6a0bb797d50d2f0d8ff04f147d0c1c7ac7f8 not found: ID does not exist" Jul 28 17:15:22.839943 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.839879 2572 scope.go:117] "RemoveContainer" containerID="983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e" Jul 28 17:15:22.840081 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.840065 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e"} err="failed to get container status \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": rpc error: code = NotFound desc = could not find container \"983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e\": container with ID starting with 983c0395826316084589a872b8c0fbec551ba56af99a717cdff76876b84f275e not found: ID does not exist" Jul 28 17:15:22.840081 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.840080 2572 scope.go:117] "RemoveContainer" containerID="cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542" Jul 28 17:15:22.840281 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.840265 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542"} err="failed to get container status \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": rpc error: code = NotFound desc = could not find container \"cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542\": container with ID starting with cd6807f64ee4ab16fa4c200c828fc3b9f2334df2d581ae61dd915add5efd6542 not found: ID does not exist" Jul 28 17:15:22.857126 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857107 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:15:22.857457 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857441 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="init-config-reloader" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857459 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="init-config-reloader" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857483 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-web" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857492 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-web" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857509 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857518 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857531 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="config-reloader" Jul 28 17:15:22.857538 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857538 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="config-reloader" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857547 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-thanos" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857555 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-thanos" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857565 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="thanos-sidecar" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857573 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="thanos-sidecar" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857593 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="prometheus" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857601 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="prometheus" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857676 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="thanos-sidecar" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857686 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857694 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-thanos" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857706 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="config-reloader" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857714 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="prometheus" Jul 28 17:15:22.857895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.857722 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" containerName="kube-rbac-proxy-web" Jul 28 17:15:22.863298 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.863276 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.865757 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.867812 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.868133 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.868134 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.868428 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.868646 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.868839 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.869024 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.869253 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c282c4kfpdeev\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.869273 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.869552 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wm4hv\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.869632 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:15:22.869738 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.869557 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:15:22.871620 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.871336 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:15:22.873436 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.873409 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:15:22.876014 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.875996 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:15:22.880328 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.880309 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:15:22.995717 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkff\" (UniqueName: \"kubernetes.io/projected/508f9394-e01c-40bb-aef6-fe8b2754d93c-kube-api-access-tjkff\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995818 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.995967 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995954 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-web-config\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.995972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/508f9394-e01c-40bb-aef6-fe8b2754d93c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996071 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996101 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-config\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/508f9394-e01c-40bb-aef6-fe8b2754d93c-config-out\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996146 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:22.996288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:22.996246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097068 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097205 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097205 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkff\" (UniqueName: \"kubernetes.io/projected/508f9394-e01c-40bb-aef6-fe8b2754d93c-kube-api-access-tjkff\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097205 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097425 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-web-config\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/508f9394-e01c-40bb-aef6-fe8b2754d93c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097618 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097587 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097960 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097626 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-config\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097960 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/508f9394-e01c-40bb-aef6-fe8b2754d93c-config-out\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097960 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.097960 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.097730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.098255 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.098230 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.098438 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.098417 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.098609 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.098704 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.099236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.099614 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.100135 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.100160 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.100301 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.099626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.100687 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.101252 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.101225 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/508f9394-e01c-40bb-aef6-fe8b2754d93c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.101378 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.101270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/508f9394-e01c-40bb-aef6-fe8b2754d93c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.102016 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.101983 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.102102 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.101990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.102211 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.102194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/508f9394-e01c-40bb-aef6-fe8b2754d93c-config-out\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.102517 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.102500 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-config\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.102780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.102757 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.103082 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.103064 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.103122 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.103091 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/508f9394-e01c-40bb-aef6-fe8b2754d93c-web-config\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.108889 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.108873 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkff\" (UniqueName: \"kubernetes.io/projected/508f9394-e01c-40bb-aef6-fe8b2754d93c-kube-api-access-tjkff\") pod \"prometheus-k8s-0\" (UID: \"508f9394-e01c-40bb-aef6-fe8b2754d93c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.175426 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.175382 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:23.316988 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.316962 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:15:23.319148 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:15:23.319124 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f9394_e01c_40bb_aef6_fe8b2754d93c.slice/crio-24537bce8423aaa7d7b2b6ac4397b8b922782dcfb1f01f803b50ba8cc0873513 WatchSource:0}: Error finding container 24537bce8423aaa7d7b2b6ac4397b8b922782dcfb1f01f803b50ba8cc0873513: Status 404 returned error can't find the container with id 24537bce8423aaa7d7b2b6ac4397b8b922782dcfb1f01f803b50ba8cc0873513 Jul 28 17:15:23.338059 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.338029 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3457fa-3f26-4551-aa80-ff5772c035c4" path="/var/lib/kubelet/pods/7c3457fa-3f26-4551-aa80-ff5772c035c4/volumes" Jul 28 17:15:23.788127 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.788090 2572 generic.go:356] "Generic (PLEG): container finished" podID="508f9394-e01c-40bb-aef6-fe8b2754d93c" containerID="1de857407417797918a420c83c6be5e47a1cd3dbafe7a0efa50bce5df338a43b" exitCode=0 Jul 28 17:15:23.788286 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.788133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerDied","Data":"1de857407417797918a420c83c6be5e47a1cd3dbafe7a0efa50bce5df338a43b"} Jul 28 17:15:23.788286 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:23.788197 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"24537bce8423aaa7d7b2b6ac4397b8b922782dcfb1f01f803b50ba8cc0873513"} Jul 28 17:15:24.795375 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.795340 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"d18a19642ea506bf31c9604511a6ad8c85c8191c0a80f600c43e28a25a52701d"} Jul 28 17:15:24.795747 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.795472 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"3727ed7ba12cee26a02a1e6611da90d40c8cc414da3164e2bdac9821ec1a67f2"} Jul 28 17:15:24.795747 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.795517 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"80fc3dddcd5d8f5c9507e6e7c8cc9dd06a18ba003ae29b8216077be6c4cc8948"} Jul 28 17:15:24.795747 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.795560 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"f9274a4701c1b867ad0b0af600114b173ee6daf25a5129dc2638005dee19faa6"} Jul 28 17:15:24.795747 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.795598 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"c17aa41419e4c60a1313d7457f7d9af984bc2d6bd0788150f023cff9fbda82a2"} Jul 28 17:15:24.795747 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.795632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"508f9394-e01c-40bb-aef6-fe8b2754d93c","Type":"ContainerStarted","Data":"2f900d8e291eb7c512c44f8b6032168521cc79ed60791cd0e8a99cea605db41f"} Jul 28 17:15:24.826711 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:24.826632 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.803979367 podStartE2EDuration="2.826611525s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.632158ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:24.825081649 +0000 UTC m=+402.014855054" watchObservedRunningTime="2026-07-28 17:15:24.826611525 +0000 UTC m=+402.016384911" Jul 28 17:15:26.268220 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:26.268167 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6kxfc" podUID="49d74a24-e3fd-4b88-84eb-f2fa8df30516" Jul 28 17:15:26.804294 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:26.804262 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:15:28.141230 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.141185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:15:28.143419 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.143380 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d74a24-e3fd-4b88-84eb-f2fa8df30516-metrics-tls\") pod \"dns-default-6kxfc\" (UID: \"49d74a24-e3fd-4b88-84eb-f2fa8df30516\") " pod="openshift-dns/dns-default-6kxfc" Jul 28 17:15:28.176297 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.176273 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:15:28.307697 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.307674 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zj6nk\"" Jul 28 17:15:28.315451 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.315431 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:15:28.445437 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.445411 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6kxfc"] Jul 28 17:15:28.447894 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:15:28.447866 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d74a24_e3fd_4b88_84eb_f2fa8df30516.slice/crio-20f88827de5166bc2f9779a158f883cb85d1e91a3478877637bf7c09fada3196 WatchSource:0}: Error finding container 20f88827de5166bc2f9779a158f883cb85d1e91a3478877637bf7c09fada3196: Status 404 returned error can't find the container with id 20f88827de5166bc2f9779a158f883cb85d1e91a3478877637bf7c09fada3196 Jul 28 17:15:28.813682 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:28.813651 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6kxfc" event={"ID":"49d74a24-e3fd-4b88-84eb-f2fa8df30516","Type":"ContainerStarted","Data":"20f88827de5166bc2f9779a158f883cb85d1e91a3478877637bf7c09fada3196"} Jul 28 17:15:29.276857 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:15:29.276768 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" podUID="19796901-f223-43c6-9e19-d37b707ca92e" Jul 28 17:15:29.818303 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:29.818275 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:15:30.824444 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:30.824405 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6kxfc" event={"ID":"49d74a24-e3fd-4b88-84eb-f2fa8df30516","Type":"ContainerStarted","Data":"f45da104b94eb84311761181ad8a0775effa6d9e75938a2c8ef5306f90c69eb2"} Jul 28 17:15:30.824897 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:30.824460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6kxfc" event={"ID":"49d74a24-e3fd-4b88-84eb-f2fa8df30516","Type":"ContainerStarted","Data":"ac72343f785e08a8f3a17dde6c4a032fd764c3c1876804e82d603c62f6a4eae3"} Jul 28 17:15:30.824897 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:30.824535 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:15:30.843913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:30.843869 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6kxfc" podStartSLOduration=373.115146012 podStartE2EDuration="6m14.843856471s" totalImagesPullingTime="1.728710459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:30.842844022 +0000 UTC m=+408.032617408" watchObservedRunningTime="2026-07-28 17:15:30.843856471 +0000 UTC m=+408.033629856" Jul 28 17:15:31.068139 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:31.068111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:15:31.070406 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:31.070365 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19796901-f223-43c6-9e19-d37b707ca92e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5j6qc\" (UID: \"19796901-f223-43c6-9e19-d37b707ca92e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:15:31.320678 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:31.320652 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n9frc\"" Jul 28 17:15:31.329502 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:31.329485 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" Jul 28 17:15:31.449666 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:31.449644 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc"] Jul 28 17:15:31.451522 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:15:31.451467 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19796901_f223_43c6_9e19_d37b707ca92e.slice/crio-111d4b334a7955cc941ceceb7cf2fe9c916bc09973609268e6a7345d0ff29167 WatchSource:0}: Error finding container 111d4b334a7955cc941ceceb7cf2fe9c916bc09973609268e6a7345d0ff29167: Status 404 returned error can't find the container with id 111d4b334a7955cc941ceceb7cf2fe9c916bc09973609268e6a7345d0ff29167 Jul 28 17:15:31.828968 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:31.828934 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" event={"ID":"19796901-f223-43c6-9e19-d37b707ca92e","Type":"ContainerStarted","Data":"111d4b334a7955cc941ceceb7cf2fe9c916bc09973609268e6a7345d0ff29167"} Jul 28 17:15:33.837856 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:33.837820 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" event={"ID":"19796901-f223-43c6-9e19-d37b707ca92e","Type":"ContainerStarted","Data":"004ac4d2b1c4d391e42ca76509fa7c33f3e2d1860e3e14dae14ae940ed73ffae"} Jul 28 17:15:33.855333 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:33.855282 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5j6qc" podStartSLOduration=373.516643768 podStartE2EDuration="6m14.855268687s" totalImagesPullingTime="1.338624919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:33.853201418 +0000 UTC m=+411.042974803" watchObservedRunningTime="2026-07-28 17:15:33.855268687 +0000 UTC m=+411.045042071" Jul 28 17:15:40.831054 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:15:40.831021 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6kxfc" Jul 28 17:16:23.175980 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:16:23.175888 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:16:23.192083 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:16:23.192055 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:16:24.057981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:16:24.057950 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:19:54.693595 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.693557 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-99zc2"] Jul 28 17:19:54.696965 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.696949 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-99zc2" Jul 28 17:19:54.699134 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.699113 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 28 17:19:54.699250 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.699229 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 17:19:54.699302 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.699263 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 17:19:54.699672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.699658 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q2x2w\"" Jul 28 17:19:54.704832 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.704440 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-99zc2"] Jul 28 17:19:54.837104 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.837078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c343a37-9956-4d96-b94f-ae03b3883f6a-model-cache\") pod \"s3-init-99zc2\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " pod="kserve/s3-init-99zc2" Jul 28 17:19:54.837203 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.837143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rftt\" (UniqueName: \"kubernetes.io/projected/5c343a37-9956-4d96-b94f-ae03b3883f6a-kube-api-access-4rftt\") pod \"s3-init-99zc2\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " pod="kserve/s3-init-99zc2" Jul 28 17:19:54.937875 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.937852 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rftt\" (UniqueName: \"kubernetes.io/projected/5c343a37-9956-4d96-b94f-ae03b3883f6a-kube-api-access-4rftt\") pod \"s3-init-99zc2\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " pod="kserve/s3-init-99zc2" Jul 28 17:19:54.937968 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.937896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c343a37-9956-4d96-b94f-ae03b3883f6a-model-cache\") pod \"s3-init-99zc2\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " pod="kserve/s3-init-99zc2" Jul 28 17:19:54.938222 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.938206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c343a37-9956-4d96-b94f-ae03b3883f6a-model-cache\") pod \"s3-init-99zc2\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " pod="kserve/s3-init-99zc2" Jul 28 17:19:54.945981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:54.945928 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rftt\" (UniqueName: \"kubernetes.io/projected/5c343a37-9956-4d96-b94f-ae03b3883f6a-kube-api-access-4rftt\") pod \"s3-init-99zc2\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " pod="kserve/s3-init-99zc2" Jul 28 17:19:55.015532 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:55.015507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-99zc2" Jul 28 17:19:55.142631 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:55.142608 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-99zc2"] Jul 28 17:19:55.145012 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:19:55.144974 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c343a37_9956_4d96_b94f_ae03b3883f6a.slice/crio-71cac3509c2d5a8f9967926c714db311d373cca6c1c0b66ba798b7ebfefd28fc WatchSource:0}: Error finding container 71cac3509c2d5a8f9967926c714db311d373cca6c1c0b66ba798b7ebfefd28fc: Status 404 returned error can't find the container with id 71cac3509c2d5a8f9967926c714db311d373cca6c1c0b66ba798b7ebfefd28fc Jul 28 17:19:55.146983 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:55.146965 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:19:55.860452 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:55.860416 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-99zc2" event={"ID":"5c343a37-9956-4d96-b94f-ae03b3883f6a","Type":"ContainerStarted","Data":"71cac3509c2d5a8f9967926c714db311d373cca6c1c0b66ba798b7ebfefd28fc"} Jul 28 17:19:58.876819 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:19:58.876746 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-99zc2" event={"ID":"5c343a37-9956-4d96-b94f-ae03b3883f6a","Type":"ContainerStarted","Data":"a8168f847a2cca3ccf53c5c1a14980830d6edfb758c9fc5955b05c57011db63a"} Jul 28 17:20:06.910347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:20:06.910260 2572 generic.go:356] "Generic (PLEG): container finished" podID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerID="a8168f847a2cca3ccf53c5c1a14980830d6edfb758c9fc5955b05c57011db63a" exitCode=0 Jul 28 17:20:06.910347 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:20:06.910323 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-99zc2" event={"ID":"5c343a37-9956-4d96-b94f-ae03b3883f6a","Type":"ContainerDied","Data":"a8168f847a2cca3ccf53c5c1a14980830d6edfb758c9fc5955b05c57011db63a"} Jul 28 17:20:11.933802 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:20:11.933769 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-99zc2" event={"ID":"5c343a37-9956-4d96-b94f-ae03b3883f6a","Type":"ContainerStarted","Data":"560c83ac6875af1f7e508bfacaa6deaeba4ee4d22d268d6ba685e216ef7afb3a"} Jul 28 17:20:11.951863 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:20:11.951801 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-99zc2" podStartSLOduration=1.9125862919999999 podStartE2EDuration="17.951782233s" totalImagesPullingTime="8.065600723s" totalInitContainerRuntime="7.973595218s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:11.949520621 +0000 UTC m=+689.139294007" watchObservedRunningTime="2026-07-28 17:20:11.951782233 +0000 UTC m=+689.141555619" Jul 28 17:21:17.209997 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:17.209960 2572 generic.go:356] "Generic (PLEG): container finished" podID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerID="560c83ac6875af1f7e508bfacaa6deaeba4ee4d22d268d6ba685e216ef7afb3a" exitCode=0 Jul 28 17:21:17.210518 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:17.210038 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-99zc2" event={"ID":"5c343a37-9956-4d96-b94f-ae03b3883f6a","Type":"ContainerDied","Data":"560c83ac6875af1f7e508bfacaa6deaeba4ee4d22d268d6ba685e216ef7afb3a"} Jul 28 17:21:18.357195 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.357167 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-99zc2" Jul 28 17:21:18.385267 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.385236 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c343a37-9956-4d96-b94f-ae03b3883f6a-model-cache\") pod \"5c343a37-9956-4d96-b94f-ae03b3883f6a\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " Jul 28 17:21:18.385403 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.385297 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rftt\" (UniqueName: \"kubernetes.io/projected/5c343a37-9956-4d96-b94f-ae03b3883f6a-kube-api-access-4rftt\") pod \"5c343a37-9956-4d96-b94f-ae03b3883f6a\" (UID: \"5c343a37-9956-4d96-b94f-ae03b3883f6a\") " Jul 28 17:21:18.387482 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.387451 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c343a37-9956-4d96-b94f-ae03b3883f6a-kube-api-access-4rftt" (OuterVolumeSpecName: "kube-api-access-4rftt") pod "5c343a37-9956-4d96-b94f-ae03b3883f6a" (UID: "5c343a37-9956-4d96-b94f-ae03b3883f6a"). InnerVolumeSpecName "kube-api-access-4rftt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:21:18.407294 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.407265 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c343a37-9956-4d96-b94f-ae03b3883f6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c343a37-9956-4d96-b94f-ae03b3883f6a" (UID: "5c343a37-9956-4d96-b94f-ae03b3883f6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:21:18.486123 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.486052 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c343a37-9956-4d96-b94f-ae03b3883f6a-model-cache\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:21:18.486123 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:18.486085 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rftt\" (UniqueName: \"kubernetes.io/projected/5c343a37-9956-4d96-b94f-ae03b3883f6a-kube-api-access-4rftt\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:21:19.219360 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:19.219325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-99zc2" event={"ID":"5c343a37-9956-4d96-b94f-ae03b3883f6a","Type":"ContainerDied","Data":"71cac3509c2d5a8f9967926c714db311d373cca6c1c0b66ba798b7ebfefd28fc"} Jul 28 17:21:19.219547 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:19.219370 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71cac3509c2d5a8f9967926c714db311d373cca6c1c0b66ba798b7ebfefd28fc" Jul 28 17:21:19.219547 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:19.219338 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-99zc2" Jul 28 17:21:31.906536 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.906501 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc"] Jul 28 17:21:31.906909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.906825 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerName="download-hf-model" Jul 28 17:21:31.906909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.906836 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerName="download-hf-model" Jul 28 17:21:31.906909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.906855 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerName="s3-init" Jul 28 17:21:31.906909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.906860 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerName="s3-init" Jul 28 17:21:31.907051 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.906913 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c343a37-9956-4d96-b94f-ae03b3883f6a" containerName="s3-init" Jul 28 17:21:31.910345 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.910328 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:31.912374 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.912348 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" Jul 28 17:21:31.912503 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.912458 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:21:31.912675 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.912660 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 17:21:31.913103 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.913074 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" Jul 28 17:21:31.913103 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.913095 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-895bt\"" Jul 28 17:21:31.913261 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.913107 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 17:21:31.913261 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.913120 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:21:31.923787 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.923765 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc"] Jul 28 17:21:31.995672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.995644 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:31.995808 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.995677 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdw69\" (UniqueName: \"kubernetes.io/projected/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-api-access-pdw69\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:31.995808 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.995711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:31.995808 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.995779 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:31.995934 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:31.995830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.022047 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.022016 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp"] Jul 28 17:21:32.025521 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.025502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.027735 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.027718 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Jul 28 17:21:32.027821 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.027720 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Jul 28 17:21:32.036446 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.036425 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp"] Jul 28 17:21:32.096634 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096606 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634e2466-37ed-4256-9178-dd70bcc947ba-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.096710 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.096710 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.096710 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.096710 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.096906 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096731 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.096906 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcw45\" (UniqueName: \"kubernetes.io/projected/634e2466-37ed-4256-9178-dd70bcc947ba-kube-api-access-jcw45\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.096906 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.096906 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634e2466-37ed-4256-9178-dd70bcc947ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.097109 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.096912 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdw69\" (UniqueName: \"kubernetes.io/projected/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-api-access-pdw69\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.097256 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.097231 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.097446 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.097431 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.097505 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.097445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.099227 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.099208 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.106122 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.106097 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdw69\" (UniqueName: \"kubernetes.io/projected/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-api-access-pdw69\") pod \"isvc-sklearn-graph-1-predictor-84c58bd689-f96qc\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.197966 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.197898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcw45\" (UniqueName: \"kubernetes.io/projected/634e2466-37ed-4256-9178-dd70bcc947ba-kube-api-access-jcw45\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.197966 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.197938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634e2466-37ed-4256-9178-dd70bcc947ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.198141 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.197979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634e2466-37ed-4256-9178-dd70bcc947ba-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.198141 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.197996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.198141 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.198026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.198458 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.198436 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634e2466-37ed-4256-9178-dd70bcc947ba-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.198732 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.198715 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.198801 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.198740 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.200302 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.200283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634e2466-37ed-4256-9178-dd70bcc947ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.208205 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.208179 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcw45\" (UniqueName: \"kubernetes.io/projected/634e2466-37ed-4256-9178-dd70bcc947ba-kube-api-access-jcw45\") pod \"isvc-xgboost-graph-predictor-555fc69d56-v4mmp\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.220899 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.220881 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:21:32.308981 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.308952 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6"] Jul 28 17:21:32.316231 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.316212 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.318277 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.318251 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 28 17:21:32.318381 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.318281 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 28 17:21:32.324845 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.324777 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6"] Jul 28 17:21:32.335128 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.335096 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:21:32.357358 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.357334 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc"] Jul 28 17:21:32.359053 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:21:32.359030 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200856f8_0ef2_4cfe_ba47_e838dcd88ff0.slice/crio-f3de0165f34f0873ec63ed405be223539c207202e6f68b71f6f14d9ff923f0b4 WatchSource:0}: Error finding container f3de0165f34f0873ec63ed405be223539c207202e6f68b71f6f14d9ff923f0b4: Status 404 returned error can't find the container with id f3de0165f34f0873ec63ed405be223539c207202e6f68b71f6f14d9ff923f0b4 Jul 28 17:21:32.399504 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.399480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwx7q\" (UniqueName: \"kubernetes.io/projected/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-api-access-lwx7q\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.399622 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.399518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52dbf1a2-5fec-4479-9b26-4f70ea29f129-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.399622 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.399588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.399701 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.399655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.399744 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.399722 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.464467 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.464438 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp"] Jul 28 17:21:32.500359 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.500331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.500486 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.500410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.500569 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.500547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.500631 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.500602 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwx7q\" (UniqueName: \"kubernetes.io/projected/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-api-access-lwx7q\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.500693 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.500644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52dbf1a2-5fec-4479-9b26-4f70ea29f129-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.500792 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.500760 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.501233 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.501130 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.501233 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.501219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.502979 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.502957 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52dbf1a2-5fec-4479-9b26-4f70ea29f129-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.515508 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.515487 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwx7q\" (UniqueName: \"kubernetes.io/projected/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-api-access-lwx7q\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.629280 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.629238 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:21:32.766436 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:32.766412 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6"] Jul 28 17:21:32.768577 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:21:32.768546 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dbf1a2_5fec_4479_9b26_4f70ea29f129.slice/crio-fe2b86c344abebb4cdd1f2f4c283c7eeaf163e5ccad9de95c1ff91b532213521 WatchSource:0}: Error finding container fe2b86c344abebb4cdd1f2f4c283c7eeaf163e5ccad9de95c1ff91b532213521: Status 404 returned error can't find the container with id fe2b86c344abebb4cdd1f2f4c283c7eeaf163e5ccad9de95c1ff91b532213521 Jul 28 17:21:33.274670 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:33.274623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerStarted","Data":"07852a2959401243e7fad8026df1a6867d38614613e78e47ad69fef939ea19b5"} Jul 28 17:21:33.275075 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:33.274734 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerStarted","Data":"fe2b86c344abebb4cdd1f2f4c283c7eeaf163e5ccad9de95c1ff91b532213521"} Jul 28 17:21:33.276305 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:33.276279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerStarted","Data":"fad6222813f88bc6d5720b022a26aa2f36b3eb2e9a87adc60001bc20c43b6f05"} Jul 28 17:21:33.276426 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:33.276339 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerStarted","Data":"94093d5648bf8c0362f073b244d81e9a798be99ea036207e9abaa4c7ead623c5"} Jul 28 17:21:33.277938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:33.277913 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerStarted","Data":"1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd"} Jul 28 17:21:33.278012 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:33.277970 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerStarted","Data":"f3de0165f34f0873ec63ed405be223539c207202e6f68b71f6f14d9ff923f0b4"} Jul 28 17:21:37.299489 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:37.299454 2572 generic.go:356] "Generic (PLEG): container finished" podID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerID="07852a2959401243e7fad8026df1a6867d38614613e78e47ad69fef939ea19b5" exitCode=0 Jul 28 17:21:37.299892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:37.299533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerDied","Data":"07852a2959401243e7fad8026df1a6867d38614613e78e47ad69fef939ea19b5"} Jul 28 17:21:37.301303 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:37.301177 2572 generic.go:356] "Generic (PLEG): container finished" podID="634e2466-37ed-4256-9178-dd70bcc947ba" containerID="fad6222813f88bc6d5720b022a26aa2f36b3eb2e9a87adc60001bc20c43b6f05" exitCode=0 Jul 28 17:21:37.301303 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:37.301255 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerDied","Data":"fad6222813f88bc6d5720b022a26aa2f36b3eb2e9a87adc60001bc20c43b6f05"} Jul 28 17:21:37.302849 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:37.302830 2572 generic.go:356] "Generic (PLEG): container finished" podID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerID="1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd" exitCode=0 Jul 28 17:21:37.302927 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:21:37.302881 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerDied","Data":"1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd"} Jul 28 17:22:05.524943 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:05.524857 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerStarted","Data":"3cc2abaadf25efcc5deeefdbb3d7f75f57f6825392c191f0daef6ce2cc87e3af"} Jul 28 17:22:05.526999 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:05.526974 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerStarted","Data":"f94c52f6c1fd2ec1684d41d077243646a2bb52154b95679cf77667c239c6b76d"} Jul 28 17:22:06.534351 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:06.534316 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerStarted","Data":"e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30"} Jul 28 17:22:08.547414 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.547363 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerStarted","Data":"9024076a0b6c4580a73488e7b07c2c7938a15399f424f6f6f5f2aae832103b9a"} Jul 28 17:22:08.547978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.547611 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:22:08.547978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.547679 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:22:08.549177 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.549136 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:08.553409 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.551702 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerStarted","Data":"5f6567c1645576b8290cccb43482dd21b838845e3f18b098882eff84e8733e27"} Jul 28 17:22:08.553409 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.552608 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:22:08.553409 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.552668 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:22:08.555671 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.553761 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:08.557450 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.557418 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerStarted","Data":"ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9"} Jul 28 17:22:08.557563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.557470 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:22:08.557563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.557515 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:22:08.558434 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.558409 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:08.571117 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.571064 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podStartSLOduration=2.913275376 podStartE2EDuration="36.571046784s" totalImagesPullingTime="30.066038039s" totalInitContainerRuntime="3.591733369s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:08.568470951 +0000 UTC m=+805.758244337" watchObservedRunningTime="2026-07-28 17:22:08.571046784 +0000 UTC m=+805.760820172" Jul 28 17:22:08.587222 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.587176 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podStartSLOduration=3.69652659 podStartE2EDuration="37.587159628s" totalImagesPullingTime="30.018248409s" totalInitContainerRuntime="3.872384629s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:08.585549301 +0000 UTC m=+805.775322697" watchObservedRunningTime="2026-07-28 17:22:08.587159628 +0000 UTC m=+805.776933018" Jul 28 17:22:08.602697 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:08.602657 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podStartSLOduration=2.691321175 podStartE2EDuration="36.602644038s" totalImagesPullingTime="30.059607182s" totalInitContainerRuntime="3.851715681s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:08.601969657 +0000 UTC m=+805.791743042" watchObservedRunningTime="2026-07-28 17:22:08.602644038 +0000 UTC m=+805.792417422" Jul 28 17:22:09.562500 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:09.562459 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:09.562909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:09.562699 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:09.562909 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:09.562706 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:10.567432 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:10.567361 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:14.568075 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:14.568039 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:22:14.569340 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:14.568828 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:14.569340 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:14.568941 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:22:14.569492 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:14.569370 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:15.572621 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:15.572589 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:22:15.573309 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:15.573272 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:24.569369 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:24.569319 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:24.569881 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:24.569474 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:25.573639 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:25.573594 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:34.569447 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:34.569375 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:34.569840 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:34.569484 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:35.573915 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:35.573863 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:44.568936 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:44.568894 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:44.569535 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:44.569503 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:45.573211 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:45.573162 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:22:54.569133 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:54.569086 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:22:54.569562 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:54.569435 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:22:55.573800 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:22:55.573754 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:23:04.569027 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:04.568980 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:23:04.569559 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:04.569528 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:23:05.573548 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:05.573518 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:23:14.569761 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:14.569714 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:23:14.569761 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:14.569733 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:23:24.570104 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:24.570073 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:23:24.570549 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:23:24.570169 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:24:01.931522 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:01.931491 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc"] Jul 28 17:24:01.932028 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:01.931730 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" containerID="cri-o://e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30" gracePeriod=30 Jul 28 17:24:01.932028 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:01.931779 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kube-rbac-proxy" containerID="cri-o://ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9" gracePeriod=30 Jul 28 17:24:01.987654 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:01.987624 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp"] Jul 28 17:24:01.987901 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:01.987877 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" containerID="cri-o://3cc2abaadf25efcc5deeefdbb3d7f75f57f6825392c191f0daef6ce2cc87e3af" gracePeriod=30 Jul 28 17:24:01.987980 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:01.987906 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kube-rbac-proxy" containerID="cri-o://5f6567c1645576b8290cccb43482dd21b838845e3f18b098882eff84e8733e27" gracePeriod=30 Jul 28 17:24:02.071558 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:02.071520 2572 generic.go:356] "Generic (PLEG): container finished" podID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerID="ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9" exitCode=2 Jul 28 17:24:02.071727 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:02.071568 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerDied","Data":"ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9"} Jul 28 17:24:02.155238 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:02.155199 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6"] Jul 28 17:24:02.155513 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:02.155473 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" containerID="cri-o://f94c52f6c1fd2ec1684d41d077243646a2bb52154b95679cf77667c239c6b76d" gracePeriod=30 Jul 28 17:24:02.155585 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:02.155504 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kube-rbac-proxy" containerID="cri-o://9024076a0b6c4580a73488e7b07c2c7938a15399f424f6f6f5f2aae832103b9a" gracePeriod=30 Jul 28 17:24:03.078954 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:03.078916 2572 generic.go:356] "Generic (PLEG): container finished" podID="634e2466-37ed-4256-9178-dd70bcc947ba" containerID="5f6567c1645576b8290cccb43482dd21b838845e3f18b098882eff84e8733e27" exitCode=2 Jul 28 17:24:03.079377 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:03.078972 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerDied","Data":"5f6567c1645576b8290cccb43482dd21b838845e3f18b098882eff84e8733e27"} Jul 28 17:24:03.080978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:03.080954 2572 generic.go:356] "Generic (PLEG): container finished" podID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerID="9024076a0b6c4580a73488e7b07c2c7938a15399f424f6f6f5f2aae832103b9a" exitCode=2 Jul 28 17:24:03.081087 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:03.081001 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerDied","Data":"9024076a0b6c4580a73488e7b07c2c7938a15399f424f6f6f5f2aae832103b9a"} Jul 28 17:24:04.562998 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:04.562957 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused" Jul 28 17:24:04.563384 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:04.562961 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Jul 28 17:24:04.569382 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:04.569346 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 28 17:24:04.569592 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:04.569562 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Jul 28 17:24:05.567882 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:05.567831 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Jul 28 17:24:05.573219 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:05.573172 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Jul 28 17:24:06.096840 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.096807 2572 generic.go:356] "Generic (PLEG): container finished" podID="634e2466-37ed-4256-9178-dd70bcc947ba" containerID="3cc2abaadf25efcc5deeefdbb3d7f75f57f6825392c191f0daef6ce2cc87e3af" exitCode=0 Jul 28 17:24:06.096978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.096882 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerDied","Data":"3cc2abaadf25efcc5deeefdbb3d7f75f57f6825392c191f0daef6ce2cc87e3af"} Jul 28 17:24:06.134856 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.134836 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:24:06.195245 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195215 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-kube-rbac-proxy-sar-config\") pod \"634e2466-37ed-4256-9178-dd70bcc947ba\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " Jul 28 17:24:06.195363 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195261 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634e2466-37ed-4256-9178-dd70bcc947ba-proxy-tls\") pod \"634e2466-37ed-4256-9178-dd70bcc947ba\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " Jul 28 17:24:06.195363 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195298 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcw45\" (UniqueName: \"kubernetes.io/projected/634e2466-37ed-4256-9178-dd70bcc947ba-kube-api-access-jcw45\") pod \"634e2466-37ed-4256-9178-dd70bcc947ba\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " Jul 28 17:24:06.195363 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195315 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-cabundle-cert\") pod \"634e2466-37ed-4256-9178-dd70bcc947ba\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " Jul 28 17:24:06.195363 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195350 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634e2466-37ed-4256-9178-dd70bcc947ba-kserve-provision-location\") pod \"634e2466-37ed-4256-9178-dd70bcc947ba\" (UID: \"634e2466-37ed-4256-9178-dd70bcc947ba\") " Jul 28 17:24:06.195703 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195676 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634e2466-37ed-4256-9178-dd70bcc947ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "634e2466-37ed-4256-9178-dd70bcc947ba" (UID: "634e2466-37ed-4256-9178-dd70bcc947ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:24:06.195779 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195729 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "634e2466-37ed-4256-9178-dd70bcc947ba" (UID: "634e2466-37ed-4256-9178-dd70bcc947ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:06.195779 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.195762 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "634e2466-37ed-4256-9178-dd70bcc947ba" (UID: "634e2466-37ed-4256-9178-dd70bcc947ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:06.197549 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.197516 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e2466-37ed-4256-9178-dd70bcc947ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "634e2466-37ed-4256-9178-dd70bcc947ba" (UID: "634e2466-37ed-4256-9178-dd70bcc947ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:24:06.197643 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.197608 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634e2466-37ed-4256-9178-dd70bcc947ba-kube-api-access-jcw45" (OuterVolumeSpecName: "kube-api-access-jcw45") pod "634e2466-37ed-4256-9178-dd70bcc947ba" (UID: "634e2466-37ed-4256-9178-dd70bcc947ba"). InnerVolumeSpecName "kube-api-access-jcw45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:24:06.296729 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.296706 2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.296729 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.296729 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634e2466-37ed-4256-9178-dd70bcc947ba-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.296868 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.296739 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcw45\" (UniqueName: \"kubernetes.io/projected/634e2466-37ed-4256-9178-dd70bcc947ba-kube-api-access-jcw45\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.296868 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.296748 2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634e2466-37ed-4256-9178-dd70bcc947ba-cabundle-cert\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.296868 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.296757 2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634e2466-37ed-4256-9178-dd70bcc947ba-kserve-provision-location\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.774422 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.774382 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:24:06.800876 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.800849 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-proxy-tls\") pod \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " Jul 28 17:24:06.801016 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.800908 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-rbac-proxy-sar-config\") pod \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " Jul 28 17:24:06.801016 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.800955 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kserve-provision-location\") pod \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " Jul 28 17:24:06.801016 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.800975 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-cabundle-cert\") pod \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " Jul 28 17:24:06.801186 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.801024 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdw69\" (UniqueName: \"kubernetes.io/projected/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-api-access-pdw69\") pod \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\" (UID: \"200856f8-0ef2-4cfe-ba47-e838dcd88ff0\") " Jul 28 17:24:06.801285 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.801263 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "200856f8-0ef2-4cfe-ba47-e838dcd88ff0" (UID: "200856f8-0ef2-4cfe-ba47-e838dcd88ff0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:24:06.801357 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.801287 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "200856f8-0ef2-4cfe-ba47-e838dcd88ff0" (UID: "200856f8-0ef2-4cfe-ba47-e838dcd88ff0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:06.801433 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.801369 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "200856f8-0ef2-4cfe-ba47-e838dcd88ff0" (UID: "200856f8-0ef2-4cfe-ba47-e838dcd88ff0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:06.802896 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.802872 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "200856f8-0ef2-4cfe-ba47-e838dcd88ff0" (UID: "200856f8-0ef2-4cfe-ba47-e838dcd88ff0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:24:06.803008 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.802960 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-api-access-pdw69" (OuterVolumeSpecName: "kube-api-access-pdw69") pod "200856f8-0ef2-4cfe-ba47-e838dcd88ff0" (UID: "200856f8-0ef2-4cfe-ba47-e838dcd88ff0"). InnerVolumeSpecName "kube-api-access-pdw69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:24:06.901683 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.901665 2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.901683 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.901685 2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kserve-provision-location\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.901815 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.901694 2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-cabundle-cert\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.901815 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.901703 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdw69\" (UniqueName: \"kubernetes.io/projected/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-kube-api-access-pdw69\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:06.901815 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:06.901712 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/200856f8-0ef2-4cfe-ba47-e838dcd88ff0-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:07.102707 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.102674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" event={"ID":"634e2466-37ed-4256-9178-dd70bcc947ba","Type":"ContainerDied","Data":"94093d5648bf8c0362f073b244d81e9a798be99ea036207e9abaa4c7ead623c5"} Jul 28 17:24:07.102823 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.102756 2572 scope.go:117] "RemoveContainer" containerID="5f6567c1645576b8290cccb43482dd21b838845e3f18b098882eff84e8733e27" Jul 28 17:24:07.102823 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.102688 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp" Jul 28 17:24:07.104640 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.104618 2572 generic.go:356] "Generic (PLEG): container finished" podID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerID="e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30" exitCode=0 Jul 28 17:24:07.104733 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.104674 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" Jul 28 17:24:07.104733 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.104705 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerDied","Data":"e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30"} Jul 28 17:24:07.104841 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.104763 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc" event={"ID":"200856f8-0ef2-4cfe-ba47-e838dcd88ff0","Type":"ContainerDied","Data":"f3de0165f34f0873ec63ed405be223539c207202e6f68b71f6f14d9ff923f0b4"} Jul 28 17:24:07.107171 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.107153 2572 generic.go:356] "Generic (PLEG): container finished" podID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerID="f94c52f6c1fd2ec1684d41d077243646a2bb52154b95679cf77667c239c6b76d" exitCode=0 Jul 28 17:24:07.107259 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.107190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerDied","Data":"f94c52f6c1fd2ec1684d41d077243646a2bb52154b95679cf77667c239c6b76d"} Jul 28 17:24:07.113159 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.113140 2572 scope.go:117] "RemoveContainer" containerID="3cc2abaadf25efcc5deeefdbb3d7f75f57f6825392c191f0daef6ce2cc87e3af" Jul 28 17:24:07.121083 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.121070 2572 scope.go:117] "RemoveContainer" containerID="fad6222813f88bc6d5720b022a26aa2f36b3eb2e9a87adc60001bc20c43b6f05" Jul 28 17:24:07.127842 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.127822 2572 scope.go:117] "RemoveContainer" containerID="ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9" Jul 28 17:24:07.131249 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.131230 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc"] Jul 28 17:24:07.134942 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.134924 2572 scope.go:117] "RemoveContainer" containerID="e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30" Jul 28 17:24:07.137574 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.137555 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-f96qc"] Jul 28 17:24:07.142029 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.142013 2572 scope.go:117] "RemoveContainer" containerID="1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd" Jul 28 17:24:07.149604 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.149590 2572 scope.go:117] "RemoveContainer" containerID="ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9" Jul 28 17:24:07.149877 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:24:07.149856 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9\": container with ID starting with ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9 not found: ID does not exist" containerID="ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9" Jul 28 17:24:07.149960 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.149882 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9"} err="failed to get container status \"ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9\": rpc error: code = NotFound desc = could not find container \"ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9\": container with ID starting with ff6aacf0fa559a7ed195c8e872368f8770c05500be7bf9709dd9c0091830d4a9 not found: ID does not exist" Jul 28 17:24:07.149960 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.149899 2572 scope.go:117] "RemoveContainer" containerID="e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30" Jul 28 17:24:07.150118 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.150101 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp"] Jul 28 17:24:07.150160 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:24:07.150111 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30\": container with ID starting with e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30 not found: ID does not exist" containerID="e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30" Jul 28 17:24:07.150160 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.150131 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30"} err="failed to get container status \"e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30\": rpc error: code = NotFound desc = could not find container \"e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30\": container with ID starting with e79d2309e5dfe157e1a710897ccbbe017198774acf5914e28ca2785dad9f9a30 not found: ID does not exist" Jul 28 17:24:07.150160 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.150147 2572 scope.go:117] "RemoveContainer" containerID="1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd" Jul 28 17:24:07.150384 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:24:07.150369 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd\": container with ID starting with 1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd not found: ID does not exist" containerID="1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd" Jul 28 17:24:07.150441 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.150403 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd"} err="failed to get container status \"1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd\": rpc error: code = NotFound desc = could not find container \"1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd\": container with ID starting with 1bc6bf69c65aab4199bd21cfd740eb89cc6e210d6220f71f08a3d3a34d7988cd not found: ID does not exist" Jul 28 17:24:07.153792 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.153773 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-v4mmp"] Jul 28 17:24:07.336704 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.336674 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" path="/var/lib/kubelet/pods/200856f8-0ef2-4cfe-ba47-e838dcd88ff0/volumes" Jul 28 17:24:07.337414 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.337375 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" path="/var/lib/kubelet/pods/634e2466-37ed-4256-9178-dd70bcc947ba/volumes" Jul 28 17:24:07.405612 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.405594 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:24:07.507809 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.507782 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52dbf1a2-5fec-4479-9b26-4f70ea29f129-proxy-tls\") pod \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " Jul 28 17:24:07.507935 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.507822 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-cabundle-cert\") pod \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " Jul 28 17:24:07.507935 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.507860 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwx7q\" (UniqueName: \"kubernetes.io/projected/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-api-access-lwx7q\") pod \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " Jul 28 17:24:07.507935 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.507916 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-rbac-proxy-sar-config\") pod \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " Jul 28 17:24:07.508093 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.507943 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kserve-provision-location\") pod \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\" (UID: \"52dbf1a2-5fec-4479-9b26-4f70ea29f129\") " Jul 28 17:24:07.508187 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.508161 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52dbf1a2-5fec-4479-9b26-4f70ea29f129" (UID: "52dbf1a2-5fec-4479-9b26-4f70ea29f129"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:07.508253 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.508228 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52dbf1a2-5fec-4479-9b26-4f70ea29f129" (UID: "52dbf1a2-5fec-4479-9b26-4f70ea29f129"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:24:07.508288 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.508275 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52dbf1a2-5fec-4479-9b26-4f70ea29f129" (UID: "52dbf1a2-5fec-4479-9b26-4f70ea29f129"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:24:07.509796 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.509778 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52dbf1a2-5fec-4479-9b26-4f70ea29f129-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52dbf1a2-5fec-4479-9b26-4f70ea29f129" (UID: "52dbf1a2-5fec-4479-9b26-4f70ea29f129"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:24:07.509996 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.509978 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-api-access-lwx7q" (OuterVolumeSpecName: "kube-api-access-lwx7q") pod "52dbf1a2-5fec-4479-9b26-4f70ea29f129" (UID: "52dbf1a2-5fec-4479-9b26-4f70ea29f129"). InnerVolumeSpecName "kube-api-access-lwx7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:24:07.609051 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.609027 2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:07.609051 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.609047 2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kserve-provision-location\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:07.609175 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.609058 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52dbf1a2-5fec-4479-9b26-4f70ea29f129-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:07.609175 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.609067 2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52dbf1a2-5fec-4479-9b26-4f70ea29f129-cabundle-cert\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:07.609175 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:07.609075 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwx7q\" (UniqueName: \"kubernetes.io/projected/52dbf1a2-5fec-4479-9b26-4f70ea29f129-kube-api-access-lwx7q\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:24:08.113743 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.113713 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" Jul 28 17:24:08.114199 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.113747 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6" event={"ID":"52dbf1a2-5fec-4479-9b26-4f70ea29f129","Type":"ContainerDied","Data":"fe2b86c344abebb4cdd1f2f4c283c7eeaf163e5ccad9de95c1ff91b532213521"} Jul 28 17:24:08.114199 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.113821 2572 scope.go:117] "RemoveContainer" containerID="9024076a0b6c4580a73488e7b07c2c7938a15399f424f6f6f5f2aae832103b9a" Jul 28 17:24:08.123099 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.123082 2572 scope.go:117] "RemoveContainer" containerID="f94c52f6c1fd2ec1684d41d077243646a2bb52154b95679cf77667c239c6b76d" Jul 28 17:24:08.130824 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.130797 2572 scope.go:117] "RemoveContainer" containerID="07852a2959401243e7fad8026df1a6867d38614613e78e47ad69fef939ea19b5" Jul 28 17:24:08.140083 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.140062 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6"] Jul 28 17:24:08.141479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:08.141456 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-8h9k6"] Jul 28 17:24:09.336879 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:24:09.336842 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" path="/var/lib/kubelet/pods/52dbf1a2-5fec-4479-9b26-4f70ea29f129/volumes" Jul 28 17:33:17.128137 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128107 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff"] Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128427 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128444 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128457 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128463 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128473 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="storage-initializer" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128478 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="storage-initializer" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128491 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="storage-initializer" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128496 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="storage-initializer" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128503 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128509 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128514 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128520 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128526 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="storage-initializer" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128531 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="storage-initializer" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128539 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128545 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128557 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128563 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128613 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128620 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kserve-container" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128628 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52dbf1a2-5fec-4479-9b26-4f70ea29f129" containerName="kube-rbac-proxy" Jul 28 17:33:17.128626 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128636 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kserve-container" Jul 28 17:33:17.129307 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128643 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="200856f8-0ef2-4cfe-ba47-e838dcd88ff0" containerName="kube-rbac-proxy" Jul 28 17:33:17.129307 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.128648 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e2466-37ed-4256-9178-dd70bcc947ba" containerName="kube-rbac-proxy" Jul 28 17:33:17.131766 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.131750 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.135429 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.135402 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-4a23a-serving-cert\"" Jul 28 17:33:17.135540 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.135517 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:33:17.136034 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.136014 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-4a23a-kube-rbac-proxy-sar-config\"" Jul 28 17:33:17.136129 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.136057 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-895bt\"" Jul 28 17:33:17.142970 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.142945 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff"] Jul 28 17:33:17.227647 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.227618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-openshift-service-ca-bundle\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.227764 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.227653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.328076 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.328045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-openshift-service-ca-bundle\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.328222 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.328087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.328222 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:33:17.328189 2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-4a23a-serving-cert: secret "ensemble-graph-4a23a-serving-cert" not found Jul 28 17:33:17.328347 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:33:17.328239 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls podName:4615d28e-abc9-4ded-b26e-c955cb9e9c6c nodeName:}" failed. No retries permitted until 2026-07-28 17:33:17.828223216 +0000 UTC m=+1475.017996579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls") pod "ensemble-graph-4a23a-84545c9fbf-4dlff" (UID: "4615d28e-abc9-4ded-b26e-c955cb9e9c6c") : secret "ensemble-graph-4a23a-serving-cert" not found Jul 28 17:33:17.328770 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.328745 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-openshift-service-ca-bundle\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.830851 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.830816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:17.833130 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:17.833102 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls\") pod \"ensemble-graph-4a23a-84545c9fbf-4dlff\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:18.044562 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:18.044537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:18.170225 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:18.170201 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff"] Jul 28 17:33:18.172720 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:33:18.172682 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4615d28e_abc9_4ded_b26e_c955cb9e9c6c.slice/crio-b25ffda2f13291759b2c932b57c7832d688be3a1a4e43f5f293dee1cf43d77fd WatchSource:0}: Error finding container b25ffda2f13291759b2c932b57c7832d688be3a1a4e43f5f293dee1cf43d77fd: Status 404 returned error can't find the container with id b25ffda2f13291759b2c932b57c7832d688be3a1a4e43f5f293dee1cf43d77fd Jul 28 17:33:18.174553 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:18.174533 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:33:18.299272 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:18.299234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" event={"ID":"4615d28e-abc9-4ded-b26e-c955cb9e9c6c","Type":"ContainerStarted","Data":"b25ffda2f13291759b2c932b57c7832d688be3a1a4e43f5f293dee1cf43d77fd"} Jul 28 17:33:21.314697 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:21.314657 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" event={"ID":"4615d28e-abc9-4ded-b26e-c955cb9e9c6c","Type":"ContainerStarted","Data":"1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb"} Jul 28 17:33:21.315161 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:21.314828 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:21.332477 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:21.332430 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podStartSLOduration=1.954063806 podStartE2EDuration="4.332410361s" totalImagesPullingTime="2.378346555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:21.330370505 +0000 UTC m=+1478.520143903" watchObservedRunningTime="2026-07-28 17:33:21.332410361 +0000 UTC m=+1478.522183752" Jul 28 17:33:27.324507 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:27.324479 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:31.206691 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:31.206658 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff"] Jul 28 17:33:31.207139 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:31.206884 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" containerID="cri-o://1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb" gracePeriod=30 Jul 28 17:33:32.322940 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:32.322900 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:33:37.323519 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:37.323479 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:33:42.322964 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:42.322923 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:33:42.323316 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:42.323038 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:33:47.323448 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:47.323407 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:33:52.323262 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:52.323221 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:33:57.323280 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:33:57.323241 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:01.378717 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.378691 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:34:01.441059 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.441028 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls\") pod \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " Jul 28 17:34:01.441251 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.441072 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-openshift-service-ca-bundle\") pod \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\" (UID: \"4615d28e-abc9-4ded-b26e-c955cb9e9c6c\") " Jul 28 17:34:01.441519 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.441492 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4615d28e-abc9-4ded-b26e-c955cb9e9c6c" (UID: "4615d28e-abc9-4ded-b26e-c955cb9e9c6c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:34:01.443168 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.443149 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4615d28e-abc9-4ded-b26e-c955cb9e9c6c" (UID: "4615d28e-abc9-4ded-b26e-c955cb9e9c6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:34:01.467800 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.467721 2572 generic.go:356] "Generic (PLEG): container finished" podID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerID="1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb" exitCode=137 Jul 28 17:34:01.467800 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.467779 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" Jul 28 17:34:01.467985 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.467811 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" event={"ID":"4615d28e-abc9-4ded-b26e-c955cb9e9c6c","Type":"ContainerDied","Data":"1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb"} Jul 28 17:34:01.467985 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.467876 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff" event={"ID":"4615d28e-abc9-4ded-b26e-c955cb9e9c6c","Type":"ContainerDied","Data":"b25ffda2f13291759b2c932b57c7832d688be3a1a4e43f5f293dee1cf43d77fd"} Jul 28 17:34:01.467985 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.467907 2572 scope.go:117] "RemoveContainer" containerID="1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb" Jul 28 17:34:01.476676 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.476658 2572 scope.go:117] "RemoveContainer" containerID="1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb" Jul 28 17:34:01.476943 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:34:01.476922 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb\": container with ID starting with 1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb not found: ID does not exist" containerID="1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb" Jul 28 17:34:01.477016 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.476951 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb"} err="failed to get container status \"1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb\": rpc error: code = NotFound desc = could not find container \"1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb\": container with ID starting with 1c7cb265225cd18227264a4692ba5321b674e159c7073b439b2ec765c66c89eb not found: ID does not exist" Jul 28 17:34:01.487300 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.487273 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff"] Jul 28 17:34:01.491727 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.491702 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4a23a-84545c9fbf-4dlff"] Jul 28 17:34:01.541760 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.541730 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:34:01.541760 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:01.541756 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4615d28e-abc9-4ded-b26e-c955cb9e9c6c-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:34:03.341886 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:03.341851 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" path="/var/lib/kubelet/pods/4615d28e-abc9-4ded-b26e-c955cb9e9c6c/volumes" Jul 28 17:34:16.956742 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.956670 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm"] Jul 28 17:34:16.957077 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.956989 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" Jul 28 17:34:16.957077 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.957001 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" Jul 28 17:34:16.957077 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.957071 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4615d28e-abc9-4ded-b26e-c955cb9e9c6c" containerName="ensemble-graph-4a23a" Jul 28 17:34:16.959982 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.959964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:16.962216 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.962196 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ef873-kube-rbac-proxy-sar-config\"" Jul 28 17:34:16.962322 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.962241 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ef873-serving-cert\"" Jul 28 17:34:16.962384 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.962318 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-895bt\"" Jul 28 17:34:16.962384 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.962358 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:34:16.966256 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:16.966236 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm"] Jul 28 17:34:17.057478 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.057436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.057638 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.057549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2806b1-7f88-4b15-84c9-e2c422c0e049-openshift-service-ca-bundle\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.158005 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.157973 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2806b1-7f88-4b15-84c9-e2c422c0e049-openshift-service-ca-bundle\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.158154 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.158015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.158154 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:34:17.158138 2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-ef873-serving-cert: secret "sequence-graph-ef873-serving-cert" not found Jul 28 17:34:17.158230 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:34:17.158191 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls podName:8b2806b1-7f88-4b15-84c9-e2c422c0e049 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:17.658176141 +0000 UTC m=+1534.847949504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls") pod "sequence-graph-ef873-59745d8759-5ksgm" (UID: "8b2806b1-7f88-4b15-84c9-e2c422c0e049") : secret "sequence-graph-ef873-serving-cert" not found Jul 28 17:34:17.158665 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.158646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2806b1-7f88-4b15-84c9-e2c422c0e049-openshift-service-ca-bundle\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.661281 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.661244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.663570 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.663551 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls\") pod \"sequence-graph-ef873-59745d8759-5ksgm\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.870643 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.870615 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:17.999211 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:17.999186 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm"] Jul 28 17:34:18.001382 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:34:18.001353 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2806b1_7f88_4b15_84c9_e2c422c0e049.slice/crio-007ece580beb60073168d38f564946828f30f23cf8b61a3a8ff6e77fa0253772 WatchSource:0}: Error finding container 007ece580beb60073168d38f564946828f30f23cf8b61a3a8ff6e77fa0253772: Status 404 returned error can't find the container with id 007ece580beb60073168d38f564946828f30f23cf8b61a3a8ff6e77fa0253772 Jul 28 17:34:18.537481 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:18.537445 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" event={"ID":"8b2806b1-7f88-4b15-84c9-e2c422c0e049","Type":"ContainerStarted","Data":"ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0"} Jul 28 17:34:18.537672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:18.537533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" event={"ID":"8b2806b1-7f88-4b15-84c9-e2c422c0e049","Type":"ContainerStarted","Data":"007ece580beb60073168d38f564946828f30f23cf8b61a3a8ff6e77fa0253772"} Jul 28 17:34:18.537672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:18.537563 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:18.557236 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:18.557188 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podStartSLOduration=2.557167593 podStartE2EDuration="2.557167593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:18.555297665 +0000 UTC m=+1535.745071050" watchObservedRunningTime="2026-07-28 17:34:18.557167593 +0000 UTC m=+1535.746940977" Jul 28 17:34:24.545665 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:24.545634 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:27.038947 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:27.038916 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm"] Jul 28 17:34:27.039311 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:27.039108 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" containerID="cri-o://ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0" gracePeriod=30 Jul 28 17:34:29.545494 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:29.545448 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:31.456693 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.456658 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b"] Jul 28 17:34:31.460996 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.460978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.462712 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.462692 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d1fc0-kube-rbac-proxy-sar-config\"" Jul 28 17:34:31.462963 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.462945 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d1fc0-serving-cert\"" Jul 28 17:34:31.467184 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.467162 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b"] Jul 28 17:34:31.569579 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.569549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba8f91a1-631d-493e-ad7c-91ef20a9d474-proxy-tls\") pod \"ensemble-graph-d1fc0-7565fb94f7-tkz2b\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.569696 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.569597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f91a1-631d-493e-ad7c-91ef20a9d474-openshift-service-ca-bundle\") pod \"ensemble-graph-d1fc0-7565fb94f7-tkz2b\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.670576 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.670548 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba8f91a1-631d-493e-ad7c-91ef20a9d474-proxy-tls\") pod \"ensemble-graph-d1fc0-7565fb94f7-tkz2b\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.670709 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.670600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f91a1-631d-493e-ad7c-91ef20a9d474-openshift-service-ca-bundle\") pod \"ensemble-graph-d1fc0-7565fb94f7-tkz2b\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.671166 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.671147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f91a1-631d-493e-ad7c-91ef20a9d474-openshift-service-ca-bundle\") pod \"ensemble-graph-d1fc0-7565fb94f7-tkz2b\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.672869 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.672847 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba8f91a1-631d-493e-ad7c-91ef20a9d474-proxy-tls\") pod \"ensemble-graph-d1fc0-7565fb94f7-tkz2b\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.772913 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.772853 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:31.894780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:31.894758 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b"] Jul 28 17:34:31.896563 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:34:31.896537 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8f91a1_631d_493e_ad7c_91ef20a9d474.slice/crio-08a77e2cf96a1c5c34f291a67f77e234d22f60456e25fd432f651d7e2fa2eb4d WatchSource:0}: Error finding container 08a77e2cf96a1c5c34f291a67f77e234d22f60456e25fd432f651d7e2fa2eb4d: Status 404 returned error can't find the container with id 08a77e2cf96a1c5c34f291a67f77e234d22f60456e25fd432f651d7e2fa2eb4d Jul 28 17:34:32.594886 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:32.594851 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" event={"ID":"ba8f91a1-631d-493e-ad7c-91ef20a9d474","Type":"ContainerStarted","Data":"0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63"} Jul 28 17:34:32.595330 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:32.594909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" event={"ID":"ba8f91a1-631d-493e-ad7c-91ef20a9d474","Type":"ContainerStarted","Data":"08a77e2cf96a1c5c34f291a67f77e234d22f60456e25fd432f651d7e2fa2eb4d"} Jul 28 17:34:32.595330 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:32.594979 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:32.612200 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:32.612153 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podStartSLOduration=1.612139216 podStartE2EDuration="1.612139216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:32.61064891 +0000 UTC m=+1549.800422294" watchObservedRunningTime="2026-07-28 17:34:32.612139216 +0000 UTC m=+1549.801912599" Jul 28 17:34:34.545087 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:34.545045 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:38.604345 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:38.604315 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:34:39.544707 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:39.544651 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:39.544938 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:39.544814 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:44.544989 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:44.544947 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:49.544724 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:49.544686 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:54.545278 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:54.545218 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:34:57.171269 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.171245 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:57.259684 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.259663 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2806b1-7f88-4b15-84c9-e2c422c0e049-openshift-service-ca-bundle\") pod \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " Jul 28 17:34:57.259790 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.259719 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls\") pod \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\" (UID: \"8b2806b1-7f88-4b15-84c9-e2c422c0e049\") " Jul 28 17:34:57.259978 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.259958 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2806b1-7f88-4b15-84c9-e2c422c0e049-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8b2806b1-7f88-4b15-84c9-e2c422c0e049" (UID: "8b2806b1-7f88-4b15-84c9-e2c422c0e049"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:34:57.261687 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.261667 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b2806b1-7f88-4b15-84c9-e2c422c0e049" (UID: "8b2806b1-7f88-4b15-84c9-e2c422c0e049"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:34:57.360128 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.360107 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2806b1-7f88-4b15-84c9-e2c422c0e049-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:34:57.360216 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.360129 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b2806b1-7f88-4b15-84c9-e2c422c0e049-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:34:57.697270 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.697171 2572 generic.go:356] "Generic (PLEG): container finished" podID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerID="ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0" exitCode=0 Jul 28 17:34:57.697270 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.697219 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" Jul 28 17:34:57.697270 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.697247 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" event={"ID":"8b2806b1-7f88-4b15-84c9-e2c422c0e049","Type":"ContainerDied","Data":"ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0"} Jul 28 17:34:57.697490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.697290 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm" event={"ID":"8b2806b1-7f88-4b15-84c9-e2c422c0e049","Type":"ContainerDied","Data":"007ece580beb60073168d38f564946828f30f23cf8b61a3a8ff6e77fa0253772"} Jul 28 17:34:57.697490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.697310 2572 scope.go:117] "RemoveContainer" containerID="ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0" Jul 28 17:34:57.705588 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.705574 2572 scope.go:117] "RemoveContainer" containerID="ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0" Jul 28 17:34:57.705858 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:34:57.705841 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0\": container with ID starting with ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0 not found: ID does not exist" containerID="ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0" Jul 28 17:34:57.705917 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.705872 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0"} err="failed to get container status \"ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0\": rpc error: code = NotFound desc = could not find container \"ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0\": container with ID starting with ec0191c8c2715368f53871d2343b6eee1ec3c90bc2a4dddf3623b59abcfb7de0 not found: ID does not exist" Jul 28 17:34:57.715737 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.715717 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm"] Jul 28 17:34:57.721282 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:57.721260 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ef873-59745d8759-5ksgm"] Jul 28 17:34:59.337142 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:34:59.337105 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" path="/var/lib/kubelet/pods/8b2806b1-7f88-4b15-84c9-e2c422c0e049/volumes" Jul 28 17:35:27.323292 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.323259 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf"] Jul 28 17:35:27.323790 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.323740 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" Jul 28 17:35:27.323790 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.323761 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" Jul 28 17:35:27.323904 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.323866 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b2806b1-7f88-4b15-84c9-e2c422c0e049" containerName="sequence-graph-ef873" Jul 28 17:35:27.326924 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.326906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.328819 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.328798 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a513a-serving-cert\"" Jul 28 17:35:27.328915 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.328840 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a513a-kube-rbac-proxy-sar-config\"" Jul 28 17:35:27.338563 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.338540 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf"] Jul 28 17:35:27.385791 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.385763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d52c5df-884d-4976-b689-b7bf15b73269-openshift-service-ca-bundle\") pod \"sequence-graph-a513a-76cfc9c946-chvjf\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.385791 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.385795 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d52c5df-884d-4976-b689-b7bf15b73269-proxy-tls\") pod \"sequence-graph-a513a-76cfc9c946-chvjf\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.487190 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.487152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d52c5df-884d-4976-b689-b7bf15b73269-openshift-service-ca-bundle\") pod \"sequence-graph-a513a-76cfc9c946-chvjf\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.487190 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.487196 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d52c5df-884d-4976-b689-b7bf15b73269-proxy-tls\") pod \"sequence-graph-a513a-76cfc9c946-chvjf\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.487861 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.487831 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d52c5df-884d-4976-b689-b7bf15b73269-openshift-service-ca-bundle\") pod \"sequence-graph-a513a-76cfc9c946-chvjf\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.489600 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.489583 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d52c5df-884d-4976-b689-b7bf15b73269-proxy-tls\") pod \"sequence-graph-a513a-76cfc9c946-chvjf\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.638886 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.638795 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:27.767542 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.767494 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf"] Jul 28 17:35:27.772580 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:35:27.772467 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d52c5df_884d_4976_b689_b7bf15b73269.slice/crio-5c5a2eba5387322217dc5815f211d2bf72663de4dba06c4a86bc8d1334174090 WatchSource:0}: Error finding container 5c5a2eba5387322217dc5815f211d2bf72663de4dba06c4a86bc8d1334174090: Status 404 returned error can't find the container with id 5c5a2eba5387322217dc5815f211d2bf72663de4dba06c4a86bc8d1334174090 Jul 28 17:35:27.825807 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:27.825784 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" event={"ID":"0d52c5df-884d-4976-b689-b7bf15b73269","Type":"ContainerStarted","Data":"5c5a2eba5387322217dc5815f211d2bf72663de4dba06c4a86bc8d1334174090"} Jul 28 17:35:28.831920 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:28.831886 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" event={"ID":"0d52c5df-884d-4976-b689-b7bf15b73269","Type":"ContainerStarted","Data":"39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b"} Jul 28 17:35:28.832284 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:28.832011 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:35:28.848463 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:28.848419 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podStartSLOduration=1.848384979 podStartE2EDuration="1.848384979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:28.846845467 +0000 UTC m=+1606.036618849" watchObservedRunningTime="2026-07-28 17:35:28.848384979 +0000 UTC m=+1606.038158363" Jul 28 17:35:34.842196 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:35:34.842167 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:42:46.055035 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:42:46.054998 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b"] Jul 28 17:42:46.057615 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:42:46.055269 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" containerID="cri-o://0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63" gracePeriod=30 Jul 28 17:42:48.604690 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:42:48.604652 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:42:53.603014 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:42:53.602967 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:42:58.602714 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:42:58.602668 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:42:58.603179 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:42:58.602835 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:43:03.603153 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:03.603113 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:08.603713 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:08.603632 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:13.603609 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:13.603559 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:16.693455 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.693429 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:43:16.700587 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700565 2572 generic.go:356] "Generic (PLEG): container finished" podID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerID="0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63" exitCode=0 Jul 28 17:43:16.700700 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700597 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" Jul 28 17:43:16.700700 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700603 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" event={"ID":"ba8f91a1-631d-493e-ad7c-91ef20a9d474","Type":"ContainerDied","Data":"0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63"} Jul 28 17:43:16.700700 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700634 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba8f91a1-631d-493e-ad7c-91ef20a9d474-proxy-tls\") pod \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " Jul 28 17:43:16.700700 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700642 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b" event={"ID":"ba8f91a1-631d-493e-ad7c-91ef20a9d474","Type":"ContainerDied","Data":"08a77e2cf96a1c5c34f291a67f77e234d22f60456e25fd432f651d7e2fa2eb4d"} Jul 28 17:43:16.700700 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700665 2572 scope.go:117] "RemoveContainer" containerID="0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63" Jul 28 17:43:16.700880 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.700709 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f91a1-631d-493e-ad7c-91ef20a9d474-openshift-service-ca-bundle\") pod \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\" (UID: \"ba8f91a1-631d-493e-ad7c-91ef20a9d474\") " Jul 28 17:43:16.701093 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.701073 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8f91a1-631d-493e-ad7c-91ef20a9d474-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ba8f91a1-631d-493e-ad7c-91ef20a9d474" (UID: "ba8f91a1-631d-493e-ad7c-91ef20a9d474"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:43:16.702734 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.702711 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8f91a1-631d-493e-ad7c-91ef20a9d474-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba8f91a1-631d-493e-ad7c-91ef20a9d474" (UID: "ba8f91a1-631d-493e-ad7c-91ef20a9d474"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:43:16.713490 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.713470 2572 scope.go:117] "RemoveContainer" containerID="0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63" Jul 28 17:43:16.713729 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:43:16.713712 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63\": container with ID starting with 0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63 not found: ID does not exist" containerID="0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63" Jul 28 17:43:16.713788 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.713735 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63"} err="failed to get container status \"0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63\": rpc error: code = NotFound desc = could not find container \"0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63\": container with ID starting with 0ee2998976dfac79092ca5557cd691a2fb11df6adb515804c150330922cd0a63 not found: ID does not exist" Jul 28 17:43:16.801666 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.801640 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba8f91a1-631d-493e-ad7c-91ef20a9d474-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:43:16.801666 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:16.801665 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f91a1-631d-493e-ad7c-91ef20a9d474-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:43:17.023892 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:17.023864 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b"] Jul 28 17:43:17.032180 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:17.032158 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d1fc0-7565fb94f7-tkz2b"] Jul 28 17:43:17.336801 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:17.336770 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" path="/var/lib/kubelet/pods/ba8f91a1-631d-493e-ad7c-91ef20a9d474/volumes" Jul 28 17:43:41.963417 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:41.963359 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf"] Jul 28 17:43:41.963819 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:41.963701 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" containerID="cri-o://39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b" gracePeriod=30 Jul 28 17:43:44.841111 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:44.841074 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:46.314228 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.314192 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq"] Jul 28 17:43:46.314616 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.314555 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" Jul 28 17:43:46.314616 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.314567 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" Jul 28 17:43:46.314686 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.314618 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8f91a1-631d-493e-ad7c-91ef20a9d474" containerName="ensemble-graph-d1fc0" Jul 28 17:43:46.318710 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.318694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:46.321028 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.321011 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-11994-kube-rbac-proxy-sar-config\"" Jul 28 17:43:46.321096 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.321024 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-11994-serving-cert\"" Jul 28 17:43:46.330004 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.329981 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq"] Jul 28 17:43:46.441823 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.441790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a5328e-bc85-487d-b33e-24d1b95856b5-openshift-service-ca-bundle\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:46.441823 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.441828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:46.542769 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.542734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a5328e-bc85-487d-b33e-24d1b95856b5-openshift-service-ca-bundle\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:46.542943 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.542777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:46.542943 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:43:46.542903 2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-11994-serving-cert: secret "splitter-graph-11994-serving-cert" not found Jul 28 17:43:46.543036 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:43:46.542960 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls podName:30a5328e-bc85-487d-b33e-24d1b95856b5 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:47.04294362 +0000 UTC m=+2104.232716983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls") pod "splitter-graph-11994-677669c5c7-fdfzq" (UID: "30a5328e-bc85-487d-b33e-24d1b95856b5") : secret "splitter-graph-11994-serving-cert" not found Jul 28 17:43:46.543368 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:46.543350 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a5328e-bc85-487d-b33e-24d1b95856b5-openshift-service-ca-bundle\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:47.046164 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.046127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:47.048457 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.048438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls\") pod \"splitter-graph-11994-677669c5c7-fdfzq\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:47.228372 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.228336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:47.360780 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.360755 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq"] Jul 28 17:43:47.362850 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:43:47.362824 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a5328e_bc85_487d_b33e_24d1b95856b5.slice/crio-082b0f7ccd3bc60b59bf7079632292533c28e4bd85d9216ddb9800a3e75dde2e WatchSource:0}: Error finding container 082b0f7ccd3bc60b59bf7079632292533c28e4bd85d9216ddb9800a3e75dde2e: Status 404 returned error can't find the container with id 082b0f7ccd3bc60b59bf7079632292533c28e4bd85d9216ddb9800a3e75dde2e Jul 28 17:43:47.822705 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.822673 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" event={"ID":"30a5328e-bc85-487d-b33e-24d1b95856b5","Type":"ContainerStarted","Data":"4f4f3dcc76a5fd2c737f6a51b40e6059f1ba04cdb413a63147b05b6442175a9a"} Jul 28 17:43:47.822908 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.822730 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" event={"ID":"30a5328e-bc85-487d-b33e-24d1b95856b5","Type":"ContainerStarted","Data":"082b0f7ccd3bc60b59bf7079632292533c28e4bd85d9216ddb9800a3e75dde2e"} Jul 28 17:43:47.822908 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.822757 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:47.840376 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:47.840329 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podStartSLOduration=1.840314754 podStartE2EDuration="1.840314754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.839880358 +0000 UTC m=+2105.029653743" watchObservedRunningTime="2026-07-28 17:43:47.840314754 +0000 UTC m=+2105.030088138" Jul 28 17:43:49.840501 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:49.840467 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:53.832230 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:53.832204 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:43:54.841062 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:54.841020 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:54.841479 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:54.841126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:43:56.375006 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:56.374931 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq"] Jul 28 17:43:56.375410 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:56.375102 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" containerID="cri-o://4f4f3dcc76a5fd2c737f6a51b40e6059f1ba04cdb413a63147b05b6442175a9a" gracePeriod=30 Jul 28 17:43:58.831368 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:58.831331 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:43:59.839962 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:43:59.839918 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:03.832158 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:03.832119 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:04.840358 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:04.840323 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:08.831674 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:08.831626 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:08.832066 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:08.831741 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:44:09.841033 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:09.841000 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:12.113582 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.113553 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:44:12.245352 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.245268 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d52c5df-884d-4976-b689-b7bf15b73269-proxy-tls\") pod \"0d52c5df-884d-4976-b689-b7bf15b73269\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " Jul 28 17:44:12.245535 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.245414 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d52c5df-884d-4976-b689-b7bf15b73269-openshift-service-ca-bundle\") pod \"0d52c5df-884d-4976-b689-b7bf15b73269\" (UID: \"0d52c5df-884d-4976-b689-b7bf15b73269\") " Jul 28 17:44:12.245768 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.245741 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d52c5df-884d-4976-b689-b7bf15b73269-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0d52c5df-884d-4976-b689-b7bf15b73269" (UID: "0d52c5df-884d-4976-b689-b7bf15b73269"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:44:12.247302 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.247282 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d52c5df-884d-4976-b689-b7bf15b73269-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d52c5df-884d-4976-b689-b7bf15b73269" (UID: "0d52c5df-884d-4976-b689-b7bf15b73269"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:44:12.346510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.346467 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d52c5df-884d-4976-b689-b7bf15b73269-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:44:12.346510 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.346504 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d52c5df-884d-4976-b689-b7bf15b73269-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:44:12.922766 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.922730 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d52c5df-884d-4976-b689-b7bf15b73269" containerID="39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b" exitCode=0 Jul 28 17:44:12.922927 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.922776 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" Jul 28 17:44:12.922927 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.922779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" event={"ID":"0d52c5df-884d-4976-b689-b7bf15b73269","Type":"ContainerDied","Data":"39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b"} Jul 28 17:44:12.922927 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.922912 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf" event={"ID":"0d52c5df-884d-4976-b689-b7bf15b73269","Type":"ContainerDied","Data":"5c5a2eba5387322217dc5815f211d2bf72663de4dba06c4a86bc8d1334174090"} Jul 28 17:44:12.923037 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.922949 2572 scope.go:117] "RemoveContainer" containerID="39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b" Jul 28 17:44:12.933704 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.933684 2572 scope.go:117] "RemoveContainer" containerID="39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b" Jul 28 17:44:12.933969 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:44:12.933948 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b\": container with ID starting with 39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b not found: ID does not exist" containerID="39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b" Jul 28 17:44:12.934019 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.933978 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b"} err="failed to get container status \"39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b\": rpc error: code = NotFound desc = could not find container \"39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b\": container with ID starting with 39d071250f38be0c36bcba721bd4a25dce23d401f8faf3054683072330fae55b not found: ID does not exist" Jul 28 17:44:12.948006 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.947983 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf"] Jul 28 17:44:12.951560 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:12.951541 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a513a-76cfc9c946-chvjf"] Jul 28 17:44:13.336769 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:13.336736 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" path="/var/lib/kubelet/pods/0d52c5df-884d-4976-b689-b7bf15b73269/volumes" Jul 28 17:44:13.831648 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:13.831616 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:18.831656 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:18.831611 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:23.831484 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:23.831446 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:44:26.982498 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:26.982467 2572 generic.go:356] "Generic (PLEG): container finished" podID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerID="4f4f3dcc76a5fd2c737f6a51b40e6059f1ba04cdb413a63147b05b6442175a9a" exitCode=0 Jul 28 17:44:26.982846 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:26.982550 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" event={"ID":"30a5328e-bc85-487d-b33e-24d1b95856b5","Type":"ContainerDied","Data":"4f4f3dcc76a5fd2c737f6a51b40e6059f1ba04cdb413a63147b05b6442175a9a"} Jul 28 17:44:27.019984 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.019964 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:44:27.053507 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.053475 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls\") pod \"30a5328e-bc85-487d-b33e-24d1b95856b5\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " Jul 28 17:44:27.053632 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.053570 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a5328e-bc85-487d-b33e-24d1b95856b5-openshift-service-ca-bundle\") pod \"30a5328e-bc85-487d-b33e-24d1b95856b5\" (UID: \"30a5328e-bc85-487d-b33e-24d1b95856b5\") " Jul 28 17:44:27.053951 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.053921 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a5328e-bc85-487d-b33e-24d1b95856b5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "30a5328e-bc85-487d-b33e-24d1b95856b5" (UID: "30a5328e-bc85-487d-b33e-24d1b95856b5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:44:27.055441 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.055416 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30a5328e-bc85-487d-b33e-24d1b95856b5" (UID: "30a5328e-bc85-487d-b33e-24d1b95856b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:44:27.154206 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.154154 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30a5328e-bc85-487d-b33e-24d1b95856b5-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:44:27.154206 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.154175 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a5328e-bc85-487d-b33e-24d1b95856b5-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:44:27.987843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.987807 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" event={"ID":"30a5328e-bc85-487d-b33e-24d1b95856b5","Type":"ContainerDied","Data":"082b0f7ccd3bc60b59bf7079632292533c28e4bd85d9216ddb9800a3e75dde2e"} Jul 28 17:44:27.987843 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.987833 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq" Jul 28 17:44:27.988334 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:27.987884 2572 scope.go:117] "RemoveContainer" containerID="4f4f3dcc76a5fd2c737f6a51b40e6059f1ba04cdb413a63147b05b6442175a9a" Jul 28 17:44:28.010410 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:28.010375 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq"] Jul 28 17:44:28.015895 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:28.015873 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-11994-677669c5c7-fdfzq"] Jul 28 17:44:29.337148 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:29.337112 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" path="/var/lib/kubelet/pods/30a5328e-bc85-487d-b33e-24d1b95856b5/volumes" Jul 28 17:44:42.226912 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.226834 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k"] Jul 28 17:44:42.227344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.227161 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" Jul 28 17:44:42.227344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.227175 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" Jul 28 17:44:42.227344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.227212 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" Jul 28 17:44:42.227344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.227218 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" Jul 28 17:44:42.227344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.227268 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a5328e-bc85-487d-b33e-24d1b95856b5" containerName="splitter-graph-11994" Jul 28 17:44:42.227344 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.227276 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d52c5df-884d-4976-b689-b7bf15b73269" containerName="sequence-graph-a513a" Jul 28 17:44:42.231864 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.231840 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.234026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.234002 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-0dc16-kube-rbac-proxy-sar-config\"" Jul 28 17:44:42.234026 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.234018 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-895bt\"" Jul 28 17:44:42.234249 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.234007 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:44:42.234249 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.234046 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-0dc16-serving-cert\"" Jul 28 17:44:42.238242 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.238219 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k"] Jul 28 17:44:42.265488 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.265462 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.265631 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.265511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8363b843-9b61-46b5-a905-f0e27057eda5-openshift-service-ca-bundle\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.366270 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.366235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.366501 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.366292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8363b843-9b61-46b5-a905-f0e27057eda5-openshift-service-ca-bundle\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.366501 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:44:42.366431 2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-0dc16-serving-cert: secret "switch-graph-0dc16-serving-cert" not found Jul 28 17:44:42.366627 ip-10-0-132-138 kubenswrapper[2572]: E0728 17:44:42.366525 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls podName:8363b843-9b61-46b5-a905-f0e27057eda5 nodeName:}" failed. No retries permitted until 2026-07-28 17:44:42.866501162 +0000 UTC m=+2160.056274525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls") pod "switch-graph-0dc16-6959997c7c-wpt8k" (UID: "8363b843-9b61-46b5-a905-f0e27057eda5") : secret "switch-graph-0dc16-serving-cert" not found Jul 28 17:44:42.366916 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.366897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8363b843-9b61-46b5-a905-f0e27057eda5-openshift-service-ca-bundle\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.869287 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.869249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:42.871566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:42.871543 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls\") pod \"switch-graph-0dc16-6959997c7c-wpt8k\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:43.144060 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:43.143984 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:43.271537 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:43.271509 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k"] Jul 28 17:44:43.273358 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:44:43.273333 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8363b843_9b61_46b5_a905_f0e27057eda5.slice/crio-f7d6cdebd5054ecc9ca1c9c0037c5587b66255b09e27f1eb20d896d961cb554f WatchSource:0}: Error finding container f7d6cdebd5054ecc9ca1c9c0037c5587b66255b09e27f1eb20d896d961cb554f: Status 404 returned error can't find the container with id f7d6cdebd5054ecc9ca1c9c0037c5587b66255b09e27f1eb20d896d961cb554f Jul 28 17:44:44.056477 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:44.056444 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" event={"ID":"8363b843-9b61-46b5-a905-f0e27057eda5","Type":"ContainerStarted","Data":"31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88"} Jul 28 17:44:44.056672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:44.056498 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" event={"ID":"8363b843-9b61-46b5-a905-f0e27057eda5","Type":"ContainerStarted","Data":"f7d6cdebd5054ecc9ca1c9c0037c5587b66255b09e27f1eb20d896d961cb554f"} Jul 28 17:44:44.056672 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:44.056540 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:44.074613 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:44.074567 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podStartSLOduration=2.074550398 podStartE2EDuration="2.074550398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:44.073990809 +0000 UTC m=+2161.263764195" watchObservedRunningTime="2026-07-28 17:44:44.074550398 +0000 UTC m=+2161.264323782" Jul 28 17:44:50.067035 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:50.066997 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 17:44:56.643416 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.643368 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk"] Jul 28 17:44:56.646815 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.646797 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.649053 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.649028 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-824fb-serving-cert\"" Jul 28 17:44:56.649188 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.649036 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-824fb-kube-rbac-proxy-sar-config\"" Jul 28 17:44:56.657566 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.657542 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk"] Jul 28 17:44:56.775484 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.775453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-openshift-service-ca-bundle\") pod \"splitter-graph-824fb-7b8cd8fdfd-pdxxk\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.775484 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.775485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-proxy-tls\") pod \"splitter-graph-824fb-7b8cd8fdfd-pdxxk\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.876845 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.876818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-openshift-service-ca-bundle\") pod \"splitter-graph-824fb-7b8cd8fdfd-pdxxk\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.876961 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.876847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-proxy-tls\") pod \"splitter-graph-824fb-7b8cd8fdfd-pdxxk\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.877587 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.877567 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-openshift-service-ca-bundle\") pod \"splitter-graph-824fb-7b8cd8fdfd-pdxxk\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.879493 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.879471 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-proxy-tls\") pod \"splitter-graph-824fb-7b8cd8fdfd-pdxxk\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:56.957994 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:56.957936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:57.083851 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:57.083823 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk"] Jul 28 17:44:57.086554 ip-10-0-132-138 kubenswrapper[2572]: W0728 17:44:57.086521 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e5e47e_7df9_4193_bb5d_7a4aecca45ce.slice/crio-42d8d0e86e6a32786d1489f1de2c3bed3b25712df92c36ce989ee350dc8e653d WatchSource:0}: Error finding container 42d8d0e86e6a32786d1489f1de2c3bed3b25712df92c36ce989ee350dc8e653d: Status 404 returned error can't find the container with id 42d8d0e86e6a32786d1489f1de2c3bed3b25712df92c36ce989ee350dc8e653d Jul 28 17:44:57.111279 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:57.111253 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" event={"ID":"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce","Type":"ContainerStarted","Data":"42d8d0e86e6a32786d1489f1de2c3bed3b25712df92c36ce989ee350dc8e653d"} Jul 28 17:44:58.116671 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:58.116636 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" event={"ID":"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce","Type":"ContainerStarted","Data":"62c088b73c3ce49f9a95fe09758fd8ed4162c81f1d1f492c13c44c39c1a3eeaf"} Jul 28 17:44:58.117086 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:58.116728 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:44:58.132915 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:44:58.132866 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podStartSLOduration=2.132853998 podStartE2EDuration="2.132853998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:58.131429105 +0000 UTC m=+2175.321202488" watchObservedRunningTime="2026-07-28 17:44:58.132853998 +0000 UTC m=+2175.322627398" Jul 28 17:45:04.127548 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:45:04.127521 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:53:11.176152 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:11.176117 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk"] Jul 28 17:53:11.178803 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:11.176492 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" containerID="cri-o://62c088b73c3ce49f9a95fe09758fd8ed4162c81f1d1f492c13c44c39c1a3eeaf" gracePeriod=30 Jul 28 17:53:14.124849 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:14.124813 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:53:19.125471 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:19.125430 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:53:24.125546 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:24.125512 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:53:24.125955 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:24.125639 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:53:29.125619 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:29.125576 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:53:34.124613 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:34.124569 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:53:39.125541 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:39.125457 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:53:41.221685 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.221653 2572 generic.go:356] "Generic (PLEG): container finished" podID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerID="62c088b73c3ce49f9a95fe09758fd8ed4162c81f1d1f492c13c44c39c1a3eeaf" exitCode=0 Jul 28 17:53:41.222074 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.221738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" event={"ID":"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce","Type":"ContainerDied","Data":"62c088b73c3ce49f9a95fe09758fd8ed4162c81f1d1f492c13c44c39c1a3eeaf"} Jul 28 17:53:41.321177 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.321157 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:53:41.454476 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.454406 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-openshift-service-ca-bundle\") pod \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " Jul 28 17:53:41.454476 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.454460 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-proxy-tls\") pod \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\" (UID: \"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce\") " Jul 28 17:53:41.454751 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.454727 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" (UID: "d7e5e47e-7df9-4193-bb5d-7a4aecca45ce"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:53:41.456433 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.456412 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" (UID: "d7e5e47e-7df9-4193-bb5d-7a4aecca45ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:53:41.555029 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.554990 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:53:41.555029 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:41.555027 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 17:53:42.227024 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:42.226991 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" Jul 28 17:53:42.227476 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:42.226991 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk" event={"ID":"d7e5e47e-7df9-4193-bb5d-7a4aecca45ce","Type":"ContainerDied","Data":"42d8d0e86e6a32786d1489f1de2c3bed3b25712df92c36ce989ee350dc8e653d"} Jul 28 17:53:42.227476 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:42.227149 2572 scope.go:117] "RemoveContainer" containerID="62c088b73c3ce49f9a95fe09758fd8ed4162c81f1d1f492c13c44c39c1a3eeaf" Jul 28 17:53:42.253710 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:42.253684 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk"] Jul 28 17:53:42.255835 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:42.255812 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-824fb-7b8cd8fdfd-pdxxk"] Jul 28 17:53:43.336250 ip-10-0-132-138 kubenswrapper[2572]: I0728 17:53:43.336217 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" path="/var/lib/kubelet/pods/d7e5e47e-7df9-4193-bb5d-7a4aecca45ce/volumes" Jul 28 18:01:01.516479 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:01.516435 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k"] Jul 28 18:01:01.517048 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:01.516704 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" containerID="cri-o://31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88" gracePeriod=30 Jul 28 18:01:02.972679 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.972645 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6mdvh/must-gather-fjdht"] Jul 28 18:01:02.973064 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.972953 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" Jul 28 18:01:02.973064 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.972964 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" Jul 28 18:01:02.973064 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.973029 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e5e47e-7df9-4193-bb5d-7a4aecca45ce" containerName="splitter-graph-824fb" Jul 28 18:01:02.976857 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.976840 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:02.979084 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.979064 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6mdvh\"/\"kube-root-ca.crt\"" Jul 28 18:01:02.979183 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.979142 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6mdvh\"/\"openshift-service-ca.crt\"" Jul 28 18:01:02.979593 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.979574 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6mdvh\"/\"default-dockercfg-ltwqk\"" Jul 28 18:01:02.986618 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:02.986595 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6mdvh/must-gather-fjdht"] Jul 28 18:01:03.077607 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.077573 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg994\" (UniqueName: \"kubernetes.io/projected/64cb9098-5664-4250-92c6-6e6093dca25c-kube-api-access-pg994\") pod \"must-gather-fjdht\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.077792 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.077625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64cb9098-5664-4250-92c6-6e6093dca25c-must-gather-output\") pod \"must-gather-fjdht\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.178464 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.178434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg994\" (UniqueName: \"kubernetes.io/projected/64cb9098-5664-4250-92c6-6e6093dca25c-kube-api-access-pg994\") pod \"must-gather-fjdht\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.178652 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.178474 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64cb9098-5664-4250-92c6-6e6093dca25c-must-gather-output\") pod \"must-gather-fjdht\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.178842 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.178822 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64cb9098-5664-4250-92c6-6e6093dca25c-must-gather-output\") pod \"must-gather-fjdht\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.187869 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.187842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg994\" (UniqueName: \"kubernetes.io/projected/64cb9098-5664-4250-92c6-6e6093dca25c-kube-api-access-pg994\") pod \"must-gather-fjdht\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.299559 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.299500 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:03.431174 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.431149 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6mdvh/must-gather-fjdht"] Jul 28 18:01:03.433378 ip-10-0-132-138 kubenswrapper[2572]: W0728 18:01:03.433352 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64cb9098_5664_4250_92c6_6e6093dca25c.slice/crio-5377f62b208b793b458d4c6e6eb899c6b0a3e8b0a7bdb973df766eab76889201 WatchSource:0}: Error finding container 5377f62b208b793b458d4c6e6eb899c6b0a3e8b0a7bdb973df766eab76889201: Status 404 returned error can't find the container with id 5377f62b208b793b458d4c6e6eb899c6b0a3e8b0a7bdb973df766eab76889201 Jul 28 18:01:03.435213 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:03.435198 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:01:04.004152 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:04.004109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6mdvh/must-gather-fjdht" event={"ID":"64cb9098-5664-4250-92c6-6e6093dca25c","Type":"ContainerStarted","Data":"5377f62b208b793b458d4c6e6eb899c6b0a3e8b0a7bdb973df766eab76889201"} Jul 28 18:01:05.067791 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:05.067511 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:01:10.035279 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:10.035236 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6mdvh/must-gather-fjdht" event={"ID":"64cb9098-5664-4250-92c6-6e6093dca25c","Type":"ContainerStarted","Data":"72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18"} Jul 28 18:01:10.035279 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:10.035280 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6mdvh/must-gather-fjdht" event={"ID":"64cb9098-5664-4250-92c6-6e6093dca25c","Type":"ContainerStarted","Data":"ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b"} Jul 28 18:01:10.052454 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:10.052405 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6mdvh/must-gather-fjdht" podStartSLOduration=2.165075354 podStartE2EDuration="8.052374857s" totalImagesPullingTime="5.887299503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:10.051489733 +0000 UTC m=+3147.241263118" watchObservedRunningTime="2026-07-28 18:01:10.052374857 +0000 UTC m=+3147.242148241" Jul 28 18:01:10.065797 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:10.065757 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:01:15.065989 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:15.065951 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:01:15.066711 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:15.066051 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 18:01:18.055744 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:18.055715 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:18.952369 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:18.952329 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:19.813298 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:19.813262 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:20.066197 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:20.066113 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:01:20.635472 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:20.635433 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:21.464525 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:21.464490 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:22.274294 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:22.274265 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:23.095154 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:23.095129 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:23.912212 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:23.912185 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:24.778760 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:24.778730 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:25.065152 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:25.065116 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:01:25.663949 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:25.663919 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:26.519071 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:26.519037 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:27.421888 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:27.421857 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:28.311264 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:28.311229 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:29.149591 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:29.149546 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0dc16-6959997c7c-wpt8k_8363b843-9b61-46b5-a905-f0e27057eda5/switch-graph-0dc16/0.log" Jul 28 18:01:30.065003 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:30.064920 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:01:30.124459 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:30.124429 2572 generic.go:356] "Generic (PLEG): container finished" podID="64cb9098-5664-4250-92c6-6e6093dca25c" containerID="ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b" exitCode=0 Jul 28 18:01:30.124583 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:30.124475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6mdvh/must-gather-fjdht" event={"ID":"64cb9098-5664-4250-92c6-6e6093dca25c","Type":"ContainerDied","Data":"ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b"} Jul 28 18:01:30.124765 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:30.124751 2572 scope.go:117] "RemoveContainer" containerID="ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b" Jul 28 18:01:30.825887 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:30.825863 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6mdvh_must-gather-fjdht_64cb9098-5664-4250-92c6-6e6093dca25c/gather/0.log" Jul 28 18:01:31.670948 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.670921 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 18:01:31.841029 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.841004 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8363b843-9b61-46b5-a905-f0e27057eda5-openshift-service-ca-bundle\") pod \"8363b843-9b61-46b5-a905-f0e27057eda5\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " Jul 28 18:01:31.841198 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.841055 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls\") pod \"8363b843-9b61-46b5-a905-f0e27057eda5\" (UID: \"8363b843-9b61-46b5-a905-f0e27057eda5\") " Jul 28 18:01:31.841425 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.841384 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8363b843-9b61-46b5-a905-f0e27057eda5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8363b843-9b61-46b5-a905-f0e27057eda5" (UID: "8363b843-9b61-46b5-a905-f0e27057eda5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:31.843045 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.843026 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8363b843-9b61-46b5-a905-f0e27057eda5" (UID: "8363b843-9b61-46b5-a905-f0e27057eda5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:01:31.942522 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.942494 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8363b843-9b61-46b5-a905-f0e27057eda5-openshift-service-ca-bundle\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 18:01:31.942522 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:31.942517 2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8363b843-9b61-46b5-a905-f0e27057eda5-proxy-tls\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 18:01:32.134042 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.133974 2572 generic.go:356] "Generic (PLEG): container finished" podID="8363b843-9b61-46b5-a905-f0e27057eda5" containerID="31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88" exitCode=0 Jul 28 18:01:32.134042 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.134031 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" Jul 28 18:01:32.134181 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.134054 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" event={"ID":"8363b843-9b61-46b5-a905-f0e27057eda5","Type":"ContainerDied","Data":"31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88"} Jul 28 18:01:32.134181 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.134124 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k" event={"ID":"8363b843-9b61-46b5-a905-f0e27057eda5","Type":"ContainerDied","Data":"f7d6cdebd5054ecc9ca1c9c0037c5587b66255b09e27f1eb20d896d961cb554f"} Jul 28 18:01:32.134181 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.134145 2572 scope.go:117] "RemoveContainer" containerID="31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88" Jul 28 18:01:32.142452 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.142436 2572 scope.go:117] "RemoveContainer" containerID="31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88" Jul 28 18:01:32.142676 ip-10-0-132-138 kubenswrapper[2572]: E0728 18:01:32.142657 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88\": container with ID starting with 31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88 not found: ID does not exist" containerID="31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88" Jul 28 18:01:32.142718 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.142683 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88"} err="failed to get container status \"31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88\": rpc error: code = NotFound desc = could not find container \"31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88\": container with ID starting with 31a132853e5f4631217357d4e513958e06639f4e27e6a8854f59ef18d9385f88 not found: ID does not exist" Jul 28 18:01:32.154761 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.154741 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k"] Jul 28 18:01:32.159080 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:32.159060 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0dc16-6959997c7c-wpt8k"] Jul 28 18:01:33.337171 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:33.337136 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" path="/var/lib/kubelet/pods/8363b843-9b61-46b5-a905-f0e27057eda5/volumes" Jul 28 18:01:35.295609 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:35.295585 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nc9d4_0d1534dc-38b2-4881-9fd1-80795b1e4b24/global-pull-secret-syncer/0.log" Jul 28 18:01:35.413059 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:35.413029 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m2hqd_2c921956-e37e-4cb1-920c-bd7736f44fa1/konnectivity-agent/0.log" Jul 28 18:01:35.490146 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:35.490121 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-138.ec2.internal_a28b7e1bb7bab65a1df8ef56c8ebc188/haproxy/0.log" Jul 28 18:01:36.255344 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.255308 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6mdvh/must-gather-fjdht"] Jul 28 18:01:36.255533 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.255499 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6mdvh/must-gather-fjdht" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="copy" containerID="cri-o://72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18" gracePeriod=2 Jul 28 18:01:36.259911 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.259850 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6mdvh/must-gather-fjdht"] Jul 28 18:01:36.482561 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.482539 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6mdvh_must-gather-fjdht_64cb9098-5664-4250-92c6-6e6093dca25c/copy/0.log" Jul 28 18:01:36.485117 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.485101 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:36.486783 ip-10-0-132-138 kubenswrapper[2572]: E0728 18:01:36.486759 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fjdht\" is forbidden: User \"system:node:ip-10-0-132-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6mdvh\": no relationship found between node 'ip-10-0-132-138.ec2.internal' and this object" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:36.577169 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.577149 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64cb9098-5664-4250-92c6-6e6093dca25c-must-gather-output\") pod \"64cb9098-5664-4250-92c6-6e6093dca25c\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " Jul 28 18:01:36.577285 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.577209 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg994\" (UniqueName: \"kubernetes.io/projected/64cb9098-5664-4250-92c6-6e6093dca25c-kube-api-access-pg994\") pod \"64cb9098-5664-4250-92c6-6e6093dca25c\" (UID: \"64cb9098-5664-4250-92c6-6e6093dca25c\") " Jul 28 18:01:36.578765 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.578742 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cb9098-5664-4250-92c6-6e6093dca25c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "64cb9098-5664-4250-92c6-6e6093dca25c" (UID: "64cb9098-5664-4250-92c6-6e6093dca25c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:01:36.579317 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.579292 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cb9098-5664-4250-92c6-6e6093dca25c-kube-api-access-pg994" (OuterVolumeSpecName: "kube-api-access-pg994") pod "64cb9098-5664-4250-92c6-6e6093dca25c" (UID: "64cb9098-5664-4250-92c6-6e6093dca25c"). InnerVolumeSpecName "kube-api-access-pg994". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:01:36.677926 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.677902 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg994\" (UniqueName: \"kubernetes.io/projected/64cb9098-5664-4250-92c6-6e6093dca25c-kube-api-access-pg994\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 18:01:36.677926 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:36.677924 2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64cb9098-5664-4250-92c6-6e6093dca25c-must-gather-output\") on node \"ip-10-0-132-138.ec2.internal\" DevicePath \"\"" Jul 28 18:01:37.155674 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.155651 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6mdvh_must-gather-fjdht_64cb9098-5664-4250-92c6-6e6093dca25c/copy/0.log" Jul 28 18:01:37.161989 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.156366 2572 generic.go:356] "Generic (PLEG): container finished" podID="64cb9098-5664-4250-92c6-6e6093dca25c" containerID="72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18" exitCode=143 Jul 28 18:01:37.161989 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.156585 2572 scope.go:117] "RemoveContainer" containerID="72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18" Jul 28 18:01:37.161989 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.156780 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:37.163567 ip-10-0-132-138 kubenswrapper[2572]: E0728 18:01:37.163539 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fjdht\" is forbidden: User \"system:node:ip-10-0-132-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6mdvh\": no relationship found between node 'ip-10-0-132-138.ec2.internal' and this object" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:37.169656 ip-10-0-132-138 kubenswrapper[2572]: E0728 18:01:37.169631 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fjdht\" is forbidden: User \"system:node:ip-10-0-132-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6mdvh\": no relationship found between node 'ip-10-0-132-138.ec2.internal' and this object" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" pod="openshift-must-gather-6mdvh/must-gather-fjdht" Jul 28 18:01:37.170328 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.170309 2572 scope.go:117] "RemoveContainer" containerID="ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b" Jul 28 18:01:37.181526 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.181506 2572 scope.go:117] "RemoveContainer" containerID="72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18" Jul 28 18:01:37.181781 ip-10-0-132-138 kubenswrapper[2572]: E0728 18:01:37.181763 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18\": container with ID starting with 72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18 not found: ID does not exist" containerID="72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18" Jul 28 18:01:37.181848 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.181791 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18"} err="failed to get container status \"72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18\": rpc error: code = NotFound desc = could not find container \"72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18\": container with ID starting with 72df7b7d0571b6fde96d565fd614c8dfee0550047781e7a895727211ac3b7d18 not found: ID does not exist" Jul 28 18:01:37.181848 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.181815 2572 scope.go:117] "RemoveContainer" containerID="ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b" Jul 28 18:01:37.182058 ip-10-0-132-138 kubenswrapper[2572]: E0728 18:01:37.182038 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b\": container with ID starting with ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b not found: ID does not exist" containerID="ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b" Jul 28 18:01:37.182101 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.182066 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b"} err="failed to get container status \"ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b\": rpc error: code = NotFound desc = could not find container \"ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b\": container with ID starting with ecdf8e3b3d4f83751d1b2566ce0f9cdac7078f7fefbeeeb432ebbe3f277e766b not found: ID does not exist" Jul 28 18:01:37.336713 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:37.336688 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" path="/var/lib/kubelet/pods/64cb9098-5664-4250-92c6-6e6093dca25c/volumes" Jul 28 18:01:40.447184 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.447120 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-bp2dz_bc0e9bc9-8193-4d05-9bba-6b1f5d5f70d8/cluster-monitoring-operator/0.log" Jul 28 18:01:40.478040 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.478015 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zg7pm_6bc18ab9-5065-4e75-b889-fdbb998f1aeb/kube-state-metrics/0.log" Jul 28 18:01:40.504317 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.504299 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zg7pm_6bc18ab9-5065-4e75-b889-fdbb998f1aeb/kube-rbac-proxy-main/0.log" Jul 28 18:01:40.543927 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.543906 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zg7pm_6bc18ab9-5065-4e75-b889-fdbb998f1aeb/kube-rbac-proxy-self/0.log" Jul 28 18:01:40.585123 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.585106 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-769795f47b-ltlpf_f555f995-2e0f-40ba-b0da-a78275b4a87e/metrics-server/0.log" Jul 28 18:01:40.619893 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.619876 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-jcxsp_da27684e-b333-4b84-9672-0c29deb8ae54/monitoring-plugin/0.log" Jul 28 18:01:40.741022 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.740948 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jhlt_84ecc9cb-5524-46d0-b51c-31be4077f6f9/node-exporter/0.log" Jul 28 18:01:40.762503 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.762459 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jhlt_84ecc9cb-5524-46d0-b51c-31be4077f6f9/kube-rbac-proxy/0.log" Jul 28 18:01:40.785016 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.784993 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4jhlt_84ecc9cb-5524-46d0-b51c-31be4077f6f9/init-textfile/0.log" Jul 28 18:01:40.920738 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.920713 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-r42mm_10e0eb75-b3c8-428d-92d6-f27dc9829f89/kube-rbac-proxy-main/0.log" Jul 28 18:01:40.949550 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.949531 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-r42mm_10e0eb75-b3c8-428d-92d6-f27dc9829f89/kube-rbac-proxy-self/0.log" Jul 28 18:01:40.976245 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:40.976227 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-r42mm_10e0eb75-b3c8-428d-92d6-f27dc9829f89/openshift-state-metrics/0.log" Jul 28 18:01:41.026753 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.026705 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/prometheus/0.log" Jul 28 18:01:41.057375 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.057359 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/config-reloader/0.log" Jul 28 18:01:41.092690 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.092674 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/thanos-sidecar/0.log" Jul 28 18:01:41.120896 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.120873 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/kube-rbac-proxy-web/0.log" Jul 28 18:01:41.143970 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.143955 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/kube-rbac-proxy/0.log" Jul 28 18:01:41.164614 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.164592 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/kube-rbac-proxy-thanos/0.log" Jul 28 18:01:41.194209 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.194190 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_508f9394-e01c-40bb-aef6-fe8b2754d93c/init-config-reloader/0.log" Jul 28 18:01:41.224632 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.224612 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-b8l29_3bd52883-679e-45e7-91d6-164cc3d9eb21/prometheus-operator/0.log" Jul 28 18:01:41.250184 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:41.250168 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-b8l29_3bd52883-679e-45e7-91d6-164cc3d9eb21/kube-rbac-proxy/0.log" Jul 28 18:01:42.920336 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:42.920264 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-5j6qc_19796901-f223-43c6-9e19-d37b707ca92e/networking-console-plugin/0.log" Jul 28 18:01:43.979957 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.979889 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv"] Jul 28 18:01:43.980271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980190 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" Jul 28 18:01:43.980271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980200 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" Jul 28 18:01:43.980271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980213 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="copy" Jul 28 18:01:43.980271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980219 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="copy" Jul 28 18:01:43.980271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980233 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="gather" Jul 28 18:01:43.980271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980238 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="gather" Jul 28 18:01:43.980514 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980284 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8363b843-9b61-46b5-a905-f0e27057eda5" containerName="switch-graph-0dc16" Jul 28 18:01:43.980514 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980294 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="copy" Jul 28 18:01:43.980514 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.980299 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64cb9098-5664-4250-92c6-6e6093dca25c" containerName="gather" Jul 28 18:01:43.985371 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.985349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:43.987452 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.987431 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsx4t\"/\"kube-root-ca.crt\"" Jul 28 18:01:43.987993 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.987978 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsx4t\"/\"openshift-service-ca.crt\"" Jul 28 18:01:43.988041 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.988027 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nsx4t\"/\"default-dockercfg-qqmtf\"" Jul 28 18:01:43.992412 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:43.992224 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv"] Jul 28 18:01:44.029637 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.029615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-proc\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.029750 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.029670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-sys\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.029750 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.029738 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-podres\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.029869 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.029781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlz4\" (UniqueName: \"kubernetes.io/projected/a711814f-8b37-4ad6-b9c2-6943f78b42ec-kube-api-access-2vlz4\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.029869 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.029813 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-lib-modules\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130672 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-proc\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130824 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-sys\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130824 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130752 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-sys\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130824 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130756 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-proc\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130824 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-podres\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130824 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlz4\" (UniqueName: \"kubernetes.io/projected/a711814f-8b37-4ad6-b9c2-6943f78b42ec-kube-api-access-2vlz4\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130995 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-lib-modules\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130995 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-podres\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.130995 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.130970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a711814f-8b37-4ad6-b9c2-6943f78b42ec-lib-modules\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.138441 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.138416 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlz4\" (UniqueName: \"kubernetes.io/projected/a711814f-8b37-4ad6-b9c2-6943f78b42ec-kube-api-access-2vlz4\") pod \"perf-node-gather-daemonset-wg6xv\" (UID: \"a711814f-8b37-4ad6-b9c2-6943f78b42ec\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.296450 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.296433 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:44.414943 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.414912 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-x4bv6_15f02567-4b4a-4ac2-9b2b-844e91578ebb/volume-data-source-validator/0.log" Jul 28 18:01:44.423654 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:44.423630 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv"] Jul 28 18:01:44.426832 ip-10-0-132-138 kubenswrapper[2572]: W0728 18:01:44.426808 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda711814f_8b37_4ad6_b9c2_6943f78b42ec.slice/crio-55c89bbdaf2e77e45616ed43f841505a0b07254c88a65e652e49ed33d762eccb WatchSource:0}: Error finding container 55c89bbdaf2e77e45616ed43f841505a0b07254c88a65e652e49ed33d762eccb: Status 404 returned error can't find the container with id 55c89bbdaf2e77e45616ed43f841505a0b07254c88a65e652e49ed33d762eccb Jul 28 18:01:45.189003 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.188969 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" event={"ID":"a711814f-8b37-4ad6-b9c2-6943f78b42ec","Type":"ContainerStarted","Data":"179ef42d98972bf5344488c60d4354510cdf8212ff503dc52b92d34303ef51ad"} Jul 28 18:01:45.189437 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.189025 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" event={"ID":"a711814f-8b37-4ad6-b9c2-6943f78b42ec","Type":"ContainerStarted","Data":"55c89bbdaf2e77e45616ed43f841505a0b07254c88a65e652e49ed33d762eccb"} Jul 28 18:01:45.189437 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.189105 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:45.206794 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.206748 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" podStartSLOduration=2.206734031 podStartE2EDuration="2.206734031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:45.206197257 +0000 UTC m=+3182.395970663" watchObservedRunningTime="2026-07-28 18:01:45.206734031 +0000 UTC m=+3182.396507416" Jul 28 18:01:45.326775 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.326753 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6kxfc_49d74a24-e3fd-4b88-84eb-f2fa8df30516/dns/0.log" Jul 28 18:01:45.351146 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.351127 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6kxfc_49d74a24-e3fd-4b88-84eb-f2fa8df30516/kube-rbac-proxy/0.log" Jul 28 18:01:45.564583 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:45.564566 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npp2t_aca3ad49-96eb-4f52-b11e-4bea99d26f5d/dns-node-resolver/0.log" Jul 28 18:01:46.140928 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:46.140902 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tdhg2_156ff3fc-7c3f-49d5-a362-949d57777cf1/node-ca/0.log" Jul 28 18:01:46.933587 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:46.933557 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68ccfc8d67-d2tcw_cbf0c30a-2f8a-4c94-a25d-55140d894389/router/0.log" Jul 28 18:01:47.353442 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:47.353415 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4pld_8ab5e312-2589-4188-b5d1-fe77447c7e52/serve-healthcheck-canary/0.log" Jul 28 18:01:47.774451 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:47.774355 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-kgzlw_9cf606e8-a632-48cc-a9b1-01bd9200de94/insights-operator/0.log" Jul 28 18:01:47.776130 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:47.776109 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-kgzlw_9cf606e8-a632-48cc-a9b1-01bd9200de94/insights-operator/1.log" Jul 28 18:01:47.956957 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:47.956929 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-djchv_cd03507b-8eb3-40bc-ad2e-8c1c7c05473a/kube-rbac-proxy/0.log" Jul 28 18:01:47.977275 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:47.977240 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-djchv_cd03507b-8eb3-40bc-ad2e-8c1c7c05473a/exporter/0.log" Jul 28 18:01:47.997340 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:47.997316 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-djchv_cd03507b-8eb3-40bc-ad2e-8c1c7c05473a/extractor/0.log" Jul 28 18:01:50.501061 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:50.501031 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-99zc2_5c343a37-9956-4d96-b94f-ae03b3883f6a/s3-init/0.log" Jul 28 18:01:50.517935 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:50.517913 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-99zc2_5c343a37-9956-4d96-b94f-ae03b3883f6a/download-hf-model/0.log" Jul 28 18:01:51.202570 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:51.202545 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-wg6xv" Jul 28 18:01:55.196953 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:55.196918 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m_a1bb1ced-d125-4861-b579-954c9e7558ba/kube-storage-version-migrator-operator/1.log" Jul 28 18:01:55.197756 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:55.197739 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-kk72m_a1bb1ced-d125-4861-b579-954c9e7558ba/kube-storage-version-migrator-operator/0.log" Jul 28 18:01:56.154285 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.154259 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/kube-multus-additional-cni-plugins/0.log" Jul 28 18:01:56.180637 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.180612 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/egress-router-binary-copy/0.log" Jul 28 18:01:56.199297 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.199277 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/cni-plugins/0.log" Jul 28 18:01:56.225782 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.225758 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/bond-cni-plugin/0.log" Jul 28 18:01:56.244226 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.244206 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/routeoverride-cni/0.log" Jul 28 18:01:56.267790 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.267767 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/whereabouts-cni-bincopy/0.log" Jul 28 18:01:56.292042 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.292023 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gljn7_071211d6-06de-48c3-8f5e-fd7b5edc77d7/whereabouts-cni/0.log" Jul 28 18:01:56.737068 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.737045 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wm8lw_fe4eaf13-1705-4a64-9c91-efaa0ceaff09/kube-multus/0.log" Jul 28 18:01:56.860222 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.860195 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m4kwh_61deeab5-0b0e-4a34-80ea-e476ec574ee7/network-metrics-daemon/0.log" Jul 28 18:01:56.886840 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:56.886823 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m4kwh_61deeab5-0b0e-4a34-80ea-e476ec574ee7/kube-rbac-proxy/0.log" Jul 28 18:01:58.182686 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.182655 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/ovn-controller/0.log" Jul 28 18:01:58.198763 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.198742 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/ovn-acl-logging/0.log" Jul 28 18:01:58.226908 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.226889 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/kube-rbac-proxy-node/0.log" Jul 28 18:01:58.251201 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.251181 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 18:01:58.270674 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.270654 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/northd/0.log" Jul 28 18:01:58.291705 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.291684 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/nbdb/0.log" Jul 28 18:01:58.312271 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.312257 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/sbdb/0.log" Jul 28 18:01:58.356213 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:58.356193 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpnss_7940f71f-a825-4461-abb6-4f7f87919491/ovnkube-controller/0.log" Jul 28 18:01:59.349230 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:59.349204 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-wzm7p_44b20fb3-a288-49a1-bc95-a87e57017d08/check-endpoints/0.log" Jul 28 18:01:59.374473 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:01:59.374450 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-k6847_d3d690c0-0665-4922-9786-e2ab30ea50b9/network-check-target-container/0.log" Jul 28 18:02:00.453579 ip-10-0-132-138 kubenswrapper[2572]: I0728 18:02:00.453552 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-thtn7_bbb89330-d651-44e6-b2cf-2f813c42aa6b/iptables-alerter/0.log"