Sep 17 20:47:12.231114 ip-10-0-140-118 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 20:47:12.231125 ip-10-0-140-118 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 20:47:12.231135 ip-10-0-140-118 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 20:47:12.231540 ip-10-0-140-118 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 20:47:23.649171 ip-10-0-140-118 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 20:47:23.649185 ip-10-0-140-118 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot aa5fc3cf70834e84aab34e340a71a92c -- Sep 17 20:49:14.419685 ip-10-0-140-118 systemd[1]: Starting Kubernetes Kubelet... Sep 17 20:49:14.843149 ip-10-0-140-118 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 20:49:14.843149 ip-10-0-140-118 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 20:49:14.843149 ip-10-0-140-118 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 20:49:14.843149 ip-10-0-140-118 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 20:49:14.843149 ip-10-0-140-118 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 20:49:14.845079 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.845002 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 20:49:14.849143 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849128 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 20:49:14.849143 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849143 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849147 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849150 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849153 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849157 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849161 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849164 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849167 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849170 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849172 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849174 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849177 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849179 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849182 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849184 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849187 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849190 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849193 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849195 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849198 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 20:49:14.849204 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849200 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849203 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849206 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849209 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849213 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849217 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849220 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849223 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849226 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849228 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849231 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849233 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849236 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849238 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849241 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849243 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849246 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849250 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849252 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 20:49:14.849708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849254 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849257 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849259 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849262 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849264 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849266 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849269 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849271 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849273 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849276 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849279 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849281 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849283 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849286 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849289 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849292 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849295 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849297 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849300 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849302 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 20:49:14.850569 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849304 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849307 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849309 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849311 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849314 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849316 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849319 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849321 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849323 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849325 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849328 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849331 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849333 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849336 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849338 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849340 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849343 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849345 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849347 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849350 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 20:49:14.851315 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849353 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849356 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849358 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849361 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849376 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849379 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849381 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849384 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849387 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849389 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849394 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849397 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849400 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849403 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849406 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849408 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849412 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849416 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 20:49:14.851977 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.849419 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852099 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852127 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852132 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852135 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852139 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852142 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852144 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852147 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852150 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852152 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852155 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852157 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852160 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852162 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852165 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852167 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852170 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852172 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852175 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 20:49:14.852428 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852177 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852180 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852182 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852185 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852188 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852190 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852193 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852195 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852198 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852200 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852203 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852205 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852208 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852211 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852214 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852218 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852220 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852223 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852225 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852227 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 20:49:14.852909 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852230 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852234 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852238 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852240 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852243 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852245 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852248 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852250 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852253 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852256 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852259 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852263 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852266 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852270 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852272 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852275 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852277 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852280 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852282 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 20:49:14.853443 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852285 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852287 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852289 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852292 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852294 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852297 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852299 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852302 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852305 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852307 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852310 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852314 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852317 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852320 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852323 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852325 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852328 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852330 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852333 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852335 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 20:49:14.853959 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852338 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852340 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852343 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852345 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852347 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852351 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852353 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852356 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852358 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852360 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852383 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852386 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852390 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852393 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852397 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852400 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852403 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852407 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852410 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852416 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 20:49:14.854460 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852419 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852766 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852770 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852773 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852775 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852780 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852784 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852787 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852790 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852792 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852795 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852797 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852800 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852802 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852805 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852808 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852810 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852814 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852816 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852819 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 20:49:14.854962 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852821 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852824 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852827 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852829 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852831 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852834 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852836 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852839 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852841 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852843 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852846 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852849 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852851 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852854 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852856 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852859 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852861 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852864 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852866 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852869 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 20:49:14.855441 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852871 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852874 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852876 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852879 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852881 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852884 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852886 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852889 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852893 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852897 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852900 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852905 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852909 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852912 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852916 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852919 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852921 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852924 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852926 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 20:49:14.855958 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852929 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852931 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852934 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852936 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852939 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852941 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852944 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852946 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852949 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852951 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852954 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852956 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852959 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852961 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852964 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852966 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852969 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852971 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852973 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852976 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 20:49:14.856432 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852978 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852981 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852983 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852985 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852988 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852991 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852993 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852996 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.852998 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853001 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853003 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853005 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853008 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853010 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853012 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853015 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853018 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853020 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853023 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 20:49:14.856904 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853025 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853027 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853100 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853133 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853140 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853145 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853149 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853152 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853157 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853161 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853164 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853167 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853171 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853174 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853177 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853180 2579 flags.go:64] FLAG: --cgroup-root="" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853182 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853185 2579 flags.go:64] FLAG: --client-ca-file="" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853188 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853191 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853195 2579 flags.go:64] FLAG: --cluster-domain="" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853198 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853201 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853204 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 20:49:14.857377 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853207 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853210 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853213 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853216 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853219 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853222 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853225 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853228 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853231 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853234 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853237 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853240 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853243 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853246 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853249 2579 flags.go:64] FLAG: --enable-server="true" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853252 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853256 2579 flags.go:64] FLAG: --event-burst="100" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853259 2579 flags.go:64] FLAG: --event-qps="50" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853262 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853265 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853268 2579 flags.go:64] FLAG: --eviction-hard="" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853272 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853274 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853277 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853280 2579 flags.go:64] FLAG: --eviction-soft="" Sep 17 20:49:14.857960 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853283 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853286 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853289 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853292 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853295 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853297 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853300 2579 flags.go:64] FLAG: --feature-gates="" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853303 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853306 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853309 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853312 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853317 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853320 2579 flags.go:64] FLAG: --help="false" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853323 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-118.ec2.internal" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853326 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853329 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853331 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853335 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853338 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853340 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853343 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853346 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853348 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 20:49:14.858597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853351 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853354 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853357 2579 flags.go:64] FLAG: --kube-reserved="" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853360 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853379 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853384 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853386 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853389 2579 flags.go:64] FLAG: --lock-file="" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853392 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853394 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853397 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853403 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853406 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853409 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853411 2579 flags.go:64] FLAG: --logging-format="text" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853414 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853417 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853420 2579 flags.go:64] FLAG: --manifest-url="" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853427 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853431 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853438 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853442 2579 flags.go:64] FLAG: --max-pods="110" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853445 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853448 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853451 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 20:49:14.859195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853454 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853457 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853460 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853463 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853469 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853473 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853475 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853478 2579 flags.go:64] FLAG: --pod-cidr="" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853481 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853487 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853490 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853493 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853495 2579 flags.go:64] FLAG: --port="10250" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853498 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853501 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b12fa4e670963c86" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853504 2579 flags.go:64] FLAG: --qos-reserved="" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853507 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853509 2579 flags.go:64] FLAG: --register-node="true" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853512 2579 flags.go:64] FLAG: --register-with-taints="" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853516 2579 flags.go:64] FLAG: --registry-burst="10" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853519 2579 flags.go:64] FLAG: --registry-qps="5" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853522 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853524 2579 flags.go:64] FLAG: --reserved-memory="" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853528 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853531 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 20:49:14.859827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853533 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853536 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853540 2579 flags.go:64] FLAG: --runonce="false" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853545 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853548 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853550 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853553 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853556 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853558 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853561 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853564 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853567 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853569 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853572 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853575 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853577 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853580 2579 flags.go:64] FLAG: --system-cgroups="" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853583 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853588 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853591 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853593 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853597 2579 flags.go:64] FLAG: --tls-min-version="" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853600 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853602 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853605 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 20:49:14.860463 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853608 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853610 2579 flags.go:64] FLAG: --v="2" Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853614 2579 flags.go:64] FLAG: --version="false" Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853618 2579 flags.go:64] FLAG: --vmodule="" Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853622 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.853625 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853730 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853734 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853736 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853741 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853745 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853748 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853751 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853754 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853757 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853759 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853762 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853764 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853767 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853769 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853772 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853774 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 20:49:14.861060 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853776 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853779 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853781 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853784 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853786 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853790 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853793 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853796 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853799 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853802 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853805 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853807 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853810 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853812 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853815 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853818 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853820 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853823 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853825 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 20:49:14.861627 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853828 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853832 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853835 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853845 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853848 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853850 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853853 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853855 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853858 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853862 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853865 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853868 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853870 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853873 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853876 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853878 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853881 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853883 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853887 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 20:49:14.862098 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853890 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853893 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853896 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853898 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853901 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853903 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853906 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853908 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853911 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853914 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853917 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853919 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853922 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853925 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853930 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853932 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853935 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853938 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853941 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 20:49:14.862564 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853943 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853945 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853948 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853951 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853953 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853955 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853958 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853960 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853963 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853966 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853969 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853971 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853974 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853976 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853979 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853981 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853984 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853986 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853988 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853991 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 20:49:14.863129 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853993 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853996 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.853998 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.854001 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.854003 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.854006 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.854012 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.862438 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.862452 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.863349 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.863356 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863440 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863445 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863449 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863452 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 20:49:14.863626 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863455 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863458 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863461 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863464 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863467 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863470 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863472 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863475 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863478 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863481 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863483 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863486 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863488 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863491 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863494 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863496 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863499 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863501 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863503 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 20:49:14.863998 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863506 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863509 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863512 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863515 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863517 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863521 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863526 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863529 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863532 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863535 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863538 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863541 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863543 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863546 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863549 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863552 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863554 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863557 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863559 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863562 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 20:49:14.864570 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863566 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863569 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863572 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863575 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863578 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863580 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863583 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863585 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863588 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863590 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863593 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863595 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863598 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863600 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863606 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863608 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863611 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863614 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863617 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863619 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 20:49:14.865045 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863622 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863624 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863627 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863630 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863632 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863635 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863638 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863640 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863643 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863645 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863648 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863650 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863652 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863655 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863658 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863662 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863665 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863668 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863670 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 20:49:14.865535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863673 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863676 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863681 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863684 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863687 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863690 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863692 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863695 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863701 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863703 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863706 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863709 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863712 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863715 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863717 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863720 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 20:49:14.865996 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863722 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.863727 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863836 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863840 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863843 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863846 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863848 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863851 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863853 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863855 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863858 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863860 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863863 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863865 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863867 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 20:49:14.866409 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863870 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863873 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863876 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863878 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863881 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863883 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863886 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863888 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863891 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863894 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863896 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863899 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863901 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863904 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863907 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863909 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863912 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863914 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863916 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863919 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 20:49:14.866768 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863921 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863924 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863926 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863929 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863931 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863934 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863936 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863939 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863941 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863943 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863946 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863948 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863950 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863953 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863956 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863958 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863961 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863963 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863966 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863968 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 20:49:14.867235 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863971 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863974 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863976 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863979 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863982 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863984 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863987 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863990 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863992 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863995 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.863997 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864000 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864003 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864005 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864008 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864011 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864013 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864016 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864018 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864020 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 20:49:14.867753 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864024 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864027 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864029 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864031 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864035 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864039 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864042 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864045 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864047 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864049 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864052 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864055 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864057 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864060 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864062 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864065 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864071 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864074 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864076 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 20:49:14.868216 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864079 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864081 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864084 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864086 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864088 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864091 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:14.864093 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.864098 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.864188 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 20:49:14.868738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.868219 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 20:49:14.869387 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.869360 2579 server.go:1019] "Starting client certificate rotation" Sep 17 20:49:14.869494 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.869480 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 20:49:14.869527 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.869519 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 20:49:14.872315 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.872303 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 20:49:14.876864 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.876850 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 20:49:14.890147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.890129 2579 log.go:25] "Validated CRI v1 runtime API" Sep 17 20:49:14.892062 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.892047 2579 log.go:25] "Validated CRI v1 image API" Sep 17 20:49:14.892986 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.892971 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 20:49:14.897927 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.897910 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8642c55b-3542-4ce9-91eb-8bf5b21c9504:/dev/nvme0n1p4 ca397b0f-fd08-45ce-a855-b36b4cefb42d:/dev/nvme0n1p3] Sep 17 20:49:14.897977 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.897927 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 20:49:14.899594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.899575 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 20:49:14.902929 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.902822 2579 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:14.902336491 +0000 UTC m=+0.377018826 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102835 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b96c3b9a733f5dbaefa0963580261 SystemUUID:ec2b96c3-b9a7-33f5-dbae-fa0963580261 BootID:aa5fc3cf-7083-4e84-aab3-4e340a71a92c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:5c:cb:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:5c:cb:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:e9:3c:c3:bc:fb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 20:49:14.902929 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.902920 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 20:49:14.903055 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.902988 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 20:49:14.906737 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.906713 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 20:49:14.906869 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.906741 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 20:49:14.907621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.907611 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 20:49:14.907651 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.907625 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 20:49:14.907651 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.907632 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 20:49:14.908586 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.908576 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 20:49:14.908653 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.908645 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 20:49:14.913933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.913921 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 17 20:49:14.914049 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914040 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 20:49:14.914081 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914071 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 17 20:49:14.914118 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914088 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 20:49:14.914118 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914103 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 20:49:14.914118 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914111 2579 kubelet.go:398] "Adding apiserver pod source" Sep 17 20:49:14.914238 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914131 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 20:49:14.914467 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914456 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 20:49:14.914509 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.914477 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 20:49:14.916928 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.916902 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 20:49:14.920226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.920212 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 20:49:14.920273 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.920244 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 20:49:14.922087 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922075 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 20:49:14.922132 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922091 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 20:49:14.922132 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922097 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 20:49:14.922132 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922102 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 20:49:14.922132 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922110 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 20:49:14.922132 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922115 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 20:49:14.922132 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922130 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 20:49:14.922280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922137 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 20:49:14.922280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922144 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 20:49:14.922280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922149 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 20:49:14.922280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922162 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 20:49:14.922280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.922167 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 20:49:14.923139 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.923118 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 20:49:14.923139 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.923138 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 20:49:14.925902 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.925882 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 20:49:14.925965 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.925919 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 20:49:14.925965 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.925939 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 20:49:14.926743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.926730 2579 server.go:1268] "Started kubelet" Sep 17 20:49:14.926824 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.926804 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 20:49:14.926940 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.926875 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 20:49:14.927039 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.926939 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 20:49:14.928067 ip-10-0-140-118 systemd[1]: Started Kubernetes Kubelet. Sep 17 20:49:14.932820 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.932665 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 20:49:14.933306 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.933283 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 17 20:49:14.935451 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.935427 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 20:49:14.939197 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.939179 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 20:49:14.939839 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.939823 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 20:49:14.939916 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.939892 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:14.940571 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940553 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 20:49:14.940571 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940570 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 20:49:14.940798 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940607 2579 factory.go:55] Registering systemd factory Sep 17 20:49:14.940798 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940622 2579 factory.go:223] Registration of the systemd container factory successfully Sep 17 20:49:14.940798 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940652 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 17 20:49:14.940798 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940661 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 17 20:49:14.941068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940876 2579 factory.go:153] Registering CRI-O factory Sep 17 20:49:14.941068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940891 2579 factory.go:223] Registration of the crio container factory successfully Sep 17 20:49:14.941068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940940 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 20:49:14.941068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940957 2579 factory.go:103] Registering Raw factory Sep 17 20:49:14.941068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.940982 2579 manager.go:1196] Started watching for new ooms in manager Sep 17 20:49:14.941356 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.941342 2579 manager.go:349] Starting recovery of all containers Sep 17 20:49:14.942232 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.942213 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 20:49:14.950184 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.950171 2579 manager.go:354] Recovery completed Sep 17 20:49:14.951438 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.951415 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 17 20:49:14.952845 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.952818 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 20:49:14.952915 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.952854 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 20:49:14.953989 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.952793 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-118.ec2.internal.18d636ccbeaec87b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-118.ec2.internal,UID:ip-10-0-140-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-118.ec2.internal,},FirstTimestamp:2026-09-17 20:49:14.926704763 +0000 UTC m=+0.401387099,LastTimestamp:2026-09-17 20:49:14.926704763 +0000 UTC m=+0.401387099,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-118.ec2.internal,}" Sep 17 20:49:14.954434 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.954422 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:14.956147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956121 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:14.956210 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956159 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:14.956210 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956175 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:14.956709 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956692 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 20:49:14.956709 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956707 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 20:49:14.956827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956717 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:14.956827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.956722 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 17 20:49:14.957340 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.957326 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:14.957441 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.957348 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:14.957441 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.957356 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:14.958317 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.958214 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-118.ec2.internal.18d636ccc06ff5a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-118.ec2.internal,UID:ip-10-0-140-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-118.ec2.internal,},FirstTimestamp:2026-09-17 20:49:14.956141987 +0000 UTC m=+0.430824329,LastTimestamp:2026-09-17 20:49:14.956141987 +0000 UTC m=+0.430824329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-118.ec2.internal,}" Sep 17 20:49:14.958867 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.958853 2579 policy_none.go:49] "None policy: Start" Sep 17 20:49:14.958867 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.958868 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 20:49:14.958980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.958879 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 20:49:14.960787 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.960772 2579 policy_none.go:47] "Start" Sep 17 20:49:14.966289 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.966231 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-118.ec2.internal.18d636ccc07050df default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-118.ec2.internal,UID:ip-10-0-140-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-118.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-118.ec2.internal,},FirstTimestamp:2026-09-17 20:49:14.956165343 +0000 UTC m=+0.430847679,LastTimestamp:2026-09-17 20:49:14.956165343 +0000 UTC m=+0.430847679,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-118.ec2.internal,}" Sep 17 20:49:14.975283 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.975252 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwkm8" Sep 17 20:49:14.977249 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.977195 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-118.ec2.internal.18d636ccc0708e40 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-118.ec2.internal,UID:ip-10-0-140-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-118.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-118.ec2.internal,},FirstTimestamp:2026-09-17 20:49:14.956181056 +0000 UTC m=+0.430863395,LastTimestamp:2026-09-17 20:49:14.956181056 +0000 UTC m=+0.430863395,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-118.ec2.internal,}" Sep 17 20:49:14.980873 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.980850 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwkm8" Sep 17 20:49:14.988191 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.988137 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-118.ec2.internal.18d636ccc06ff5a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-118.ec2.internal.18d636ccc06ff5a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-118.ec2.internal,UID:ip-10-0-140-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-118.ec2.internal,},FirstTimestamp:2026-09-17 20:49:14.956141987 +0000 UTC m=+0.430824329,LastTimestamp:2026-09-17 20:49:14.957337261 +0000 UTC m=+0.432019596,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-118.ec2.internal,}" Sep 17 20:49:14.998802 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.998780 2579 manager.go:337] "Starting Device Plugin manager" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.998847 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.998862 2579 server.go:87] "Starting device plugin registration server" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.999153 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.999165 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.999283 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.999361 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:14.999388 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:14.999971 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 20:49:15.023558 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.000013 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.046008 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.045986 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 20:49:15.047206 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.047192 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 20:49:15.047269 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.047235 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 20:49:15.047269 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.047257 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 20:49:15.047343 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.047293 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 20:49:15.049457 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.049434 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 20:49:15.099320 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.099239 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:15.100138 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.100123 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:15.100222 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.100157 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:15.100222 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.100171 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:15.100222 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.100199 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.109267 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.109252 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.109349 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.109273 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-118.ec2.internal\": node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.138027 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.138008 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.148400 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.148359 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal"] Sep 17 20:49:15.148456 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.148437 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:15.149173 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.149157 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:15.149239 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.149186 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:15.149239 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.149195 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:15.151540 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.151526 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:15.151663 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.151648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.151761 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.151681 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:15.152202 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.152189 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:15.152284 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.152201 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:15.152284 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.152218 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:15.152284 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.152225 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:15.152284 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.152232 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:15.152284 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.152236 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:15.154484 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.154471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.154559 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.154497 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:15.155099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.155068 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:15.155099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.155096 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:15.155200 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.155108 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:15.178953 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.178936 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-118.ec2.internal\" not found" node="ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.183415 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.183401 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-118.ec2.internal\" not found" node="ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.239020 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.239000 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.339463 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.339422 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.343789 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.343766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/576eb62884aeb095e963216a64f67066-config\") pod \"kube-apiserver-proxy-ip-10-0-140-118.ec2.internal\" (UID: \"576eb62884aeb095e963216a64f67066\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.343875 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.343799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868967e77ed612cb8308ea9845864a59-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal\" (UID: \"868967e77ed612cb8308ea9845864a59\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.343875 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.343819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868967e77ed612cb8308ea9845864a59-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal\" (UID: \"868967e77ed612cb8308ea9845864a59\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.440211 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.440143 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.444424 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.444409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/576eb62884aeb095e963216a64f67066-config\") pod \"kube-apiserver-proxy-ip-10-0-140-118.ec2.internal\" (UID: \"576eb62884aeb095e963216a64f67066\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.444473 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.444431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868967e77ed612cb8308ea9845864a59-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal\" (UID: \"868967e77ed612cb8308ea9845864a59\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.444473 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.444447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868967e77ed612cb8308ea9845864a59-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal\" (UID: \"868967e77ed612cb8308ea9845864a59\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.444533 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.444516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/576eb62884aeb095e963216a64f67066-config\") pod \"kube-apiserver-proxy-ip-10-0-140-118.ec2.internal\" (UID: \"576eb62884aeb095e963216a64f67066\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.444565 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.444528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868967e77ed612cb8308ea9845864a59-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal\" (UID: \"868967e77ed612cb8308ea9845864a59\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.444565 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.444528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868967e77ed612cb8308ea9845864a59-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal\" (UID: \"868967e77ed612cb8308ea9845864a59\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.482572 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.482527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.486779 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.486762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:15.541151 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.541112 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.641699 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.641673 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.742221 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.742155 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.824177 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.824152 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 20:49:15.842965 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.842944 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.869270 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.869254 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 20:49:15.869635 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.869349 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 20:49:15.869635 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.869416 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 20:49:15.943641 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:15.943593 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:15.970007 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.969985 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 20:49:15.988508 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.988478 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:15 +0000 UTC" deadline="2028-02-11 07:36:49.993191805 +0000 UTC" Sep 17 20:49:15.988508 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:15.988505 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12274h47m34.004689991s" Sep 17 20:49:16.044013 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.043963 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:16.095480 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:16.095454 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576eb62884aeb095e963216a64f67066.slice/crio-fb34b6bd7218fa62d7f295de67cf6fc87188e3000335c5ba3d730b2f77988c6e WatchSource:0}: Error finding container fb34b6bd7218fa62d7f295de67cf6fc87188e3000335c5ba3d730b2f77988c6e: Status 404 returned error can't find the container with id fb34b6bd7218fa62d7f295de67cf6fc87188e3000335c5ba3d730b2f77988c6e Sep 17 20:49:16.095750 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:16.095733 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868967e77ed612cb8308ea9845864a59.slice/crio-b15124a153d64c58fa1bdb29b31643a18ed154f59fa82f2f6a723b94ac566598 WatchSource:0}: Error finding container b15124a153d64c58fa1bdb29b31643a18ed154f59fa82f2f6a723b94ac566598: Status 404 returned error can't find the container with id b15124a153d64c58fa1bdb29b31643a18ed154f59fa82f2f6a723b94ac566598 Sep 17 20:49:16.099782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.099764 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 20:49:16.100388 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.100354 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 20:49:16.102661 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.102643 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientMemory" Sep 17 20:49:16.102743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.102681 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 20:49:16.102743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.102698 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeHasSufficientPID" Sep 17 20:49:16.144113 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.144091 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:16.244701 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.244678 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:16.345317 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.345265 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:16.446143 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.446123 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-118.ec2.internal\" not found" Sep 17 20:49:16.450356 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.450342 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 20:49:16.540948 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.540925 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" Sep 17 20:49:16.548821 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.548804 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 20:49:16.549663 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.549652 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" Sep 17 20:49:16.561787 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.561772 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 20:49:16.914769 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.914736 2579 apiserver.go:52] "Watching apiserver" Sep 17 20:49:16.927714 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.927686 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 20:49:16.928088 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.928066 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-rtwxb","openshift-network-diagnostics/network-check-target-z556m","kube-system/konnectivity-agent-gb8f6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd","openshift-cluster-node-tuning-operator/tuned-phgq5","openshift-dns/node-resolver-pgqm7","openshift-multus/network-metrics-daemon-f8sff","openshift-network-operator/iptables-alerter-4kl45","openshift-ovn-kubernetes/ovnkube-node-mnbqp","kube-system/global-pull-secret-syncer-xqdcz","kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal","openshift-image-registry/node-ca-8vvzl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal","openshift-multus/multus-additional-cni-plugins-t72xc"] Sep 17 20:49:16.930827 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.930801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.933325 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.933161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.934797 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.934776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gpxh2\"" Sep 17 20:49:16.935524 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.935321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 20:49:16.935524 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.935358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.935524 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.935460 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.935911 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.935749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:16.936208 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.936187 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjlxj\"" Sep 17 20:49:16.936303 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.936221 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.938197 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.938177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:16.938415 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.938278 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:16.940359 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.940341 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 20:49:16.940747 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.940729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:16.940837 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.940797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:16.941027 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.941011 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.941103 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.941052 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 20:49:16.941161 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.941133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 20:49:16.941234 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.941216 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4l9rs\"" Sep 17 20:49:16.942966 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.942944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:16.945318 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.945296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rtwxb" Sep 17 20:49:16.946582 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.946555 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.946680 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.946585 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.946935 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.946920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wtrqz\"" Sep 17 20:49:16.948171 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.948150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:16.950815 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.950790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:16.952574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysctl-d\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.952679 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-sys\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.952679 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpk88\" (UniqueName: \"kubernetes.io/projected/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-kube-api-access-mpk88\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:16.952679 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952670 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 20:49:16.952838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-socket-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.952838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952733 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.952838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-modprobe-d\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.952838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwlb\" (UniqueName: \"kubernetes.io/projected/9f02e492-cf56-4deb-8665-67180c2517cf-kube-api-access-lpwlb\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.952838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-var-lib-kubelet\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d769f3fe-96c8-4c71-9890-6d16a4869ec1-konnectivity-ca\") pod \"konnectivity-agent-gb8f6\" (UID: \"d769f3fe-96c8-4c71-9890-6d16a4869ec1\") " pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-tmp-dir\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f02e492-cf56-4deb-8665-67180c2517cf-tmp\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.952991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-sys-fs\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953010 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244kc\" (UniqueName: \"kubernetes.io/projected/b347becc-88b4-42a1-a9bb-71a60030655f-kube-api-access-244kc\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-run\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-lib-modules\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953042 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 20:49:16.953095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-host\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953085 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f02e492-cf56-4deb-8665-67180c2517cf-etc-tuned\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqlj\" (UniqueName: \"kubernetes.io/projected/e85d8f30-029c-4596-a39d-101227da18aa-kube-api-access-qfqlj\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-hosts-file\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d769f3fe-96c8-4c71-9890-6d16a4869ec1-agent-certs\") pod \"konnectivity-agent-gb8f6\" (UID: \"d769f3fe-96c8-4c71-9890-6d16a4869ec1\") " pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-registration-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysconfig\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-device-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-kubernetes\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysctl-conf\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.953594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.953546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-systemd\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:16.954810 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.954784 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.955002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.954969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:16.955097 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.955019 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 20:49:16.955097 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:16.955041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:16.955230 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.955217 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 20:49:16.955302 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.955287 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pwr6\"" Sep 17 20:49:16.955815 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.955481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 20:49:16.955815 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.955496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ndhk6\"" Sep 17 20:49:16.955815 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.955608 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.957240 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.956299 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.957240 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.956439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 20:49:16.957240 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.956461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 20:49:16.957240 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.956834 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5c6c\"" Sep 17 20:49:16.957240 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.957145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:16.959958 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.959751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:16.962145 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.962124 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 20:49:16.962535 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.962516 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 20:49:16.963258 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.963236 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7ck58\"" Sep 17 20:49:16.964438 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.963768 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 20:49:16.964438 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.963782 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 20:49:16.964438 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.964043 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 20:49:16.964438 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.964094 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 20:49:16.965035 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.965018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wrn9l\"" Sep 17 20:49:16.996706 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:16.996544 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2glhs" Sep 17 20:49:17.009759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.009740 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2glhs" Sep 17 20:49:17.043722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.043698 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 20:49:17.051855 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.051809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" event={"ID":"576eb62884aeb095e963216a64f67066","Type":"ContainerStarted","Data":"fb34b6bd7218fa62d7f295de67cf6fc87188e3000335c5ba3d730b2f77988c6e"} Sep 17 20:49:17.053761 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.053761 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-sys\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" event={"ID":"868967e77ed612cb8308ea9845864a59","Type":"ContainerStarted","Data":"b15124a153d64c58fa1bdb29b31643a18ed154f59fa82f2f6a723b94ac566598"} Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-sys\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-ovn\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-env-overrides\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-cni-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.053949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-kubelet\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-conf-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.053986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-socket-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-modprobe-d\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwlb\" (UniqueName: \"kubernetes.io/projected/9f02e492-cf56-4deb-8665-67180c2517cf-kube-api-access-lpwlb\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-log-socket\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvtq\" (UniqueName: \"kubernetes.io/projected/e434119f-e83f-428a-9061-2e131afb5f4f-kube-api-access-9hvtq\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81833eee-01cf-420e-9cfb-8b6705c2db18-cni-binary-copy\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-multus-certs\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-kubelet-config\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-socket-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-var-lib-kubelet\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054208 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-modprobe-d\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f02e492-cf56-4deb-8665-67180c2517cf-tmp\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-var-lib-kubelet\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.054328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-system-cni-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-socket-dir-parent\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/387082f5-712b-4e1c-820e-66c7f71a0250-host\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-sys-fs\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244kc\" (UniqueName: \"kubernetes.io/projected/b347becc-88b4-42a1-a9bb-71a60030655f-kube-api-access-244kc\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-sys-fs\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-host\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f02e492-cf56-4deb-8665-67180c2517cf-etc-tuned\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-host\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-systemd\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-cnibin\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-netns\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-systemd-units\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-cni-bin\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-cni-netd\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-system-cni-dir\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-daemon-config\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.055226 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-etc-kubernetes\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbwc\" (UniqueName: \"kubernetes.io/projected/81833eee-01cf-420e-9cfb-8b6705c2db18-kube-api-access-5fbwc\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d769f3fe-96c8-4c71-9890-6d16a4869ec1-agent-certs\") pod \"konnectivity-agent-gb8f6\" (UID: \"d769f3fe-96c8-4c71-9890-6d16a4869ec1\") " pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-ovnkube-script-lib\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-cnibin\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054951 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.054988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-hostroot\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-dbus\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/387082f5-712b-4e1c-820e-66c7f71a0250-serviceca\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-registration-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysconfig\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-ovnkube-config\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-registration-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-kubernetes\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/077c8c39-4a85-4df3-a960-2ffd249b40f9-iptables-alerter-script\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.056002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysconfig\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-cni-multus\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysctl-d\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f662058f-fd45-4280-85de-0547407d6805-ovn-node-metrics-cert\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-kubernetes\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4swt\" (UniqueName: \"kubernetes.io/projected/387082f5-712b-4e1c-820e-66c7f71a0250-kube-api-access-n4swt\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpk88\" (UniqueName: \"kubernetes.io/projected/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-kube-api-access-mpk88\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbzxc\" (UniqueName: \"kubernetes.io/projected/077c8c39-4a85-4df3-a960-2ffd249b40f9-kube-api-access-jbzxc\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysctl-d\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d769f3fe-96c8-4c71-9890-6d16a4869ec1-konnectivity-ca\") pod \"konnectivity-agent-gb8f6\" (UID: \"d769f3fe-96c8-4c71-9890-6d16a4869ec1\") " pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-tmp-dir\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-kubelet\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-slash\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-run\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-lib-modules\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/077c8c39-4a85-4df3-a960-2ffd249b40f9-host-slash\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.056746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-k8s-cni-cncf-io\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-cni-bin\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqlj\" (UniqueName: \"kubernetes.io/projected/e85d8f30-029c-4596-a39d-101227da18aa-kube-api-access-qfqlj\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-hosts-file\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvqm\" (UniqueName: \"kubernetes.io/projected/f662058f-fd45-4280-85de-0547407d6805-kube-api-access-clvqm\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-tmp-dir\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-node-log\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.055972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-device-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-hosts-file\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-device-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysctl-conf\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d769f3fe-96c8-4c71-9890-6d16a4869ec1-konnectivity-ca\") pod \"konnectivity-agent-gb8f6\" (UID: \"d769f3fe-96c8-4c71-9890-6d16a4869ec1\") " pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:17.057621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-systemd\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-run-netns\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-systemd\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-var-lib-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-etc-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-lib-modules\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.056389 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.056482 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:49:17.556463799 +0000 UTC m=+3.031146145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-etc-sysctl-conf\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b347becc-88b4-42a1-a9bb-71a60030655f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.058574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.056642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f02e492-cf56-4deb-8665-67180c2517cf-run\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.062450 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.060207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f02e492-cf56-4deb-8665-67180c2517cf-tmp\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.062450 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.060424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f02e492-cf56-4deb-8665-67180c2517cf-etc-tuned\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.062450 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.060469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d769f3fe-96c8-4c71-9890-6d16a4869ec1-agent-certs\") pod \"konnectivity-agent-gb8f6\" (UID: \"d769f3fe-96c8-4c71-9890-6d16a4869ec1\") " pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:17.071295 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.071209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwlb\" (UniqueName: \"kubernetes.io/projected/9f02e492-cf56-4deb-8665-67180c2517cf-kube-api-access-lpwlb\") pod \"tuned-phgq5\" (UID: \"9f02e492-cf56-4deb-8665-67180c2517cf\") " pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.072304 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.072281 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:17.072437 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.072309 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:17.072437 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.072324 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:17.072437 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.072410 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:17.572367395 +0000 UTC m=+3.047049743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:17.072719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.072694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqlj\" (UniqueName: \"kubernetes.io/projected/e85d8f30-029c-4596-a39d-101227da18aa-kube-api-access-qfqlj\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:17.074930 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.074909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpk88\" (UniqueName: \"kubernetes.io/projected/fc6dd067-c1f7-44ad-9ec7-62bf9738c142-kube-api-access-mpk88\") pod \"node-resolver-pgqm7\" (UID: \"fc6dd067-c1f7-44ad-9ec7-62bf9738c142\") " pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.075002 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.074913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-244kc\" (UniqueName: \"kubernetes.io/projected/b347becc-88b4-42a1-a9bb-71a60030655f-kube-api-access-244kc\") pod \"aws-ebs-csi-driver-node-qgzkd\" (UID: \"b347becc-88b4-42a1-a9bb-71a60030655f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.157233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-systemd\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-cnibin\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-netns\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-netns\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-systemd\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-cnibin\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-systemd-units\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-cni-bin\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-cni-netd\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-systemd-units\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-system-cni-dir\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-daemon-config\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-cni-netd\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157499 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-cni-bin\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-etc-kubernetes\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-system-cni-dir\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.157537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbwc\" (UniqueName: \"kubernetes.io/projected/81833eee-01cf-420e-9cfb-8b6705c2db18-kube-api-access-5fbwc\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-etc-kubernetes\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-ovnkube-script-lib\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-cnibin\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-hostroot\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-dbus\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/387082f5-712b-4e1c-820e-66c7f71a0250-serviceca\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-ovnkube-config\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/077c8c39-4a85-4df3-a960-2ffd249b40f9-iptables-alerter-script\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-cni-multus\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f662058f-fd45-4280-85de-0547407d6805-ovn-node-metrics-cert\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4swt\" (UniqueName: \"kubernetes.io/projected/387082f5-712b-4e1c-820e-66c7f71a0250-kube-api-access-n4swt\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbzxc\" (UniqueName: \"kubernetes.io/projected/077c8c39-4a85-4df3-a960-2ffd249b40f9-kube-api-access-jbzxc\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.158367 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-kubelet\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-slash\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.157986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/077c8c39-4a85-4df3-a960-2ffd249b40f9-host-slash\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-k8s-cni-cncf-io\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-cni-bin\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvqm\" (UniqueName: \"kubernetes.io/projected/f662058f-fd45-4280-85de-0547407d6805-kube-api-access-clvqm\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-daemon-config\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-node-log\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-run-netns\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-var-lib-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-etc-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-cnibin\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-ovnkube-script-lib\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-hostroot\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.159245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/077c8c39-4a85-4df3-a960-2ffd249b40f9-host-slash\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-slash\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.158339 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-k8s-cni-cncf-io\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.158381 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:17.658366401 +0000 UTC m=+3.133048727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158397 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-cni-bin\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-dbus\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e434119f-e83f-428a-9061-2e131afb5f4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-kubelet\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-ovn\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-env-overrides\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-cni-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-kubelet\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-conf-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-log-socket\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvtq\" (UniqueName: \"kubernetes.io/projected/e434119f-e83f-428a-9061-2e131afb5f4f-kube-api-access-9hvtq\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.160099 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81833eee-01cf-420e-9cfb-8b6705c2db18-cni-binary-copy\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-multus-certs\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-kubelet-config\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-system-cni-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/387082f5-712b-4e1c-820e-66c7f71a0250-serviceca\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-socket-dir-parent\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/387082f5-712b-4e1c-820e-66c7f71a0250-host\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-socket-dir-parent\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159150 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-run-multus-certs\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/387082f5-712b-4e1c-820e-66c7f71a0250-host\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-node-log\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-log-socket\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-etc-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-cni-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-kubelet\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.160970 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-multus-conf-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-host-run-netns\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.158964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-ovnkube-config\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-kubelet-config\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/077c8c39-4a85-4df3-a960-2ffd249b40f9-iptables-alerter-script\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81833eee-01cf-420e-9cfb-8b6705c2db18-cni-binary-copy\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-var-lib-openvswitch\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-host-var-lib-cni-multus\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.159857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f662058f-fd45-4280-85de-0547407d6805-run-ovn\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.160016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e434119f-e83f-428a-9061-2e131afb5f4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.160123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81833eee-01cf-420e-9cfb-8b6705c2db18-system-cni-dir\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.161719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.160232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f662058f-fd45-4280-85de-0547407d6805-env-overrides\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.169918 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.169857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f662058f-fd45-4280-85de-0547407d6805-ovn-node-metrics-cert\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.173274 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.173223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4swt\" (UniqueName: \"kubernetes.io/projected/387082f5-712b-4e1c-820e-66c7f71a0250-kube-api-access-n4swt\") pod \"node-ca-8vvzl\" (UID: \"387082f5-712b-4e1c-820e-66c7f71a0250\") " pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.173387 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.173338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvqm\" (UniqueName: \"kubernetes.io/projected/f662058f-fd45-4280-85de-0547407d6805-kube-api-access-clvqm\") pod \"ovnkube-node-mnbqp\" (UID: \"f662058f-fd45-4280-85de-0547407d6805\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.173387 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.173348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbzxc\" (UniqueName: \"kubernetes.io/projected/077c8c39-4a85-4df3-a960-2ffd249b40f9-kube-api-access-jbzxc\") pod \"iptables-alerter-4kl45\" (UID: \"077c8c39-4a85-4df3-a960-2ffd249b40f9\") " pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.173873 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.173848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvtq\" (UniqueName: \"kubernetes.io/projected/e434119f-e83f-428a-9061-2e131afb5f4f-kube-api-access-9hvtq\") pod \"multus-additional-cni-plugins-t72xc\" (UID: \"e434119f-e83f-428a-9061-2e131afb5f4f\") " pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.173967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.173913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbwc\" (UniqueName: \"kubernetes.io/projected/81833eee-01cf-420e-9cfb-8b6705c2db18-kube-api-access-5fbwc\") pod \"multus-rtwxb\" (UID: \"81833eee-01cf-420e-9cfb-8b6705c2db18\") " pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.245349 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.245316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" Sep 17 20:49:17.252888 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.252868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phgq5" Sep 17 20:49:17.263984 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.263963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:17.272907 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.272886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgqm7" Sep 17 20:49:17.282817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.282795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rtwxb" Sep 17 20:49:17.292681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.292658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4kl45" Sep 17 20:49:17.300532 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.300510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:17.307865 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.307842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8vvzl" Sep 17 20:49:17.314619 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.314596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t72xc" Sep 17 20:49:17.360573 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.360545 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 20:49:17.562578 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.562485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:17.562720 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.562674 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:17.562771 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.562758 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:49:18.562736093 +0000 UTC m=+4.037418433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:17.663244 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.663214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:17.663439 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.663259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:17.663439 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.663382 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:17.663439 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.663422 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:17.663605 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.663448 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:17.663605 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.663462 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:17.663605 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.663500 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:18.663453653 +0000 UTC m=+4.138135978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:17.663605 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:17.663529 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:18.663507866 +0000 UTC m=+4.138190209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:17.897564 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:17.897478 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 20:49:18.013525 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:18.013489 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:17 +0000 UTC" deadline="2028-04-10 19:59:38.273629126 +0000 UTC" Sep 17 20:49:18.013525 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:18.013519 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13703h10m20.260113463s" Sep 17 20:49:18.047903 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:18.047870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:18.048053 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.048024 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:18.207455 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.207424 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd769f3fe_96c8_4c71_9890_6d16a4869ec1.slice/crio-cd38fe3ebc9236207c5f75645b4152c7f10a1eebda53f015495a5dfb3872fca5 WatchSource:0}: Error finding container cd38fe3ebc9236207c5f75645b4152c7f10a1eebda53f015495a5dfb3872fca5: Status 404 returned error can't find the container with id cd38fe3ebc9236207c5f75645b4152c7f10a1eebda53f015495a5dfb3872fca5 Sep 17 20:49:18.209512 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.209483 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077c8c39_4a85_4df3_a960_2ffd249b40f9.slice/crio-fdeb3ffd54453ba602ceefd65f9dde1253debf5cc55589b3eb9d9cc5b24910d5 WatchSource:0}: Error finding container fdeb3ffd54453ba602ceefd65f9dde1253debf5cc55589b3eb9d9cc5b24910d5: Status 404 returned error can't find the container with id fdeb3ffd54453ba602ceefd65f9dde1253debf5cc55589b3eb9d9cc5b24910d5 Sep 17 20:49:18.210990 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.210954 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf662058f_fd45_4280_85de_0547407d6805.slice/crio-daaa270aab1bcfb7917737190ff8b6cf4092114722d3af52463fec7499c1c4d2 WatchSource:0}: Error finding container daaa270aab1bcfb7917737190ff8b6cf4092114722d3af52463fec7499c1c4d2: Status 404 returned error can't find the container with id daaa270aab1bcfb7917737190ff8b6cf4092114722d3af52463fec7499c1c4d2 Sep 17 20:49:18.214112 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.214084 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode434119f_e83f_428a_9061_2e131afb5f4f.slice/crio-9c2ead063dbed11662f804402164d9752f90608531395f1718decfa26e297a39 WatchSource:0}: Error finding container 9c2ead063dbed11662f804402164d9752f90608531395f1718decfa26e297a39: Status 404 returned error can't find the container with id 9c2ead063dbed11662f804402164d9752f90608531395f1718decfa26e297a39 Sep 17 20:49:18.216206 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.216182 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6dd067_c1f7_44ad_9ec7_62bf9738c142.slice/crio-3d0f6d88061194a0055c1dd873f3b6de2457aeef271b088fca54f21435b4a366 WatchSource:0}: Error finding container 3d0f6d88061194a0055c1dd873f3b6de2457aeef271b088fca54f21435b4a366: Status 404 returned error can't find the container with id 3d0f6d88061194a0055c1dd873f3b6de2457aeef271b088fca54f21435b4a366 Sep 17 20:49:18.218070 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.218043 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb347becc_88b4_42a1_a9bb_71a60030655f.slice/crio-14a153c9bd763bfb4adefa7c4073c679e4c8de5b01a86b511e6a5461b918b662 WatchSource:0}: Error finding container 14a153c9bd763bfb4adefa7c4073c679e4c8de5b01a86b511e6a5461b918b662: Status 404 returned error can't find the container with id 14a153c9bd763bfb4adefa7c4073c679e4c8de5b01a86b511e6a5461b918b662 Sep 17 20:49:18.218667 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.218585 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f02e492_cf56_4deb_8665_67180c2517cf.slice/crio-65d9550baf6831b0337376b718853ad0e807f1b91b5e679ca4cdabf58160d127 WatchSource:0}: Error finding container 65d9550baf6831b0337376b718853ad0e807f1b91b5e679ca4cdabf58160d127: Status 404 returned error can't find the container with id 65d9550baf6831b0337376b718853ad0e807f1b91b5e679ca4cdabf58160d127 Sep 17 20:49:18.219840 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.219812 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387082f5_712b_4e1c_820e_66c7f71a0250.slice/crio-4c22953967f8cb1cf8ca446bf9bf8abdd01ff5dd395dd5eedce8169af9de6e4b WatchSource:0}: Error finding container 4c22953967f8cb1cf8ca446bf9bf8abdd01ff5dd395dd5eedce8169af9de6e4b: Status 404 returned error can't find the container with id 4c22953967f8cb1cf8ca446bf9bf8abdd01ff5dd395dd5eedce8169af9de6e4b Sep 17 20:49:18.220970 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:18.220945 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81833eee_01cf_420e_9cfb_8b6705c2db18.slice/crio-4c475fcec9e4df2fb3bf01bf65dc9e60c72d968f41dda28416bbd57f2ca0315b WatchSource:0}: Error finding container 4c475fcec9e4df2fb3bf01bf65dc9e60c72d968f41dda28416bbd57f2ca0315b: Status 404 returned error can't find the container with id 4c475fcec9e4df2fb3bf01bf65dc9e60c72d968f41dda28416bbd57f2ca0315b Sep 17 20:49:18.569786 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:18.569755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:18.569909 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.569876 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:18.569963 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.569932 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:49:20.569917136 +0000 UTC m=+6.044599460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:18.670744 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:18.670706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:18.670926 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:18.670773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:18.670989 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.670969 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:18.671046 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.671039 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:20.671019111 +0000 UTC m=+6.145701459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:18.671254 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.671227 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:18.671254 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.671252 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:18.671432 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.671266 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:18.671432 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:18.671338 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:20.671322341 +0000 UTC m=+6.146004668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:19.014125 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.014021 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:17 +0000 UTC" deadline="2028-06-16 01:28:04.085963938 +0000 UTC" Sep 17 20:49:19.014125 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.014080 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15292h38m45.071889164s" Sep 17 20:49:19.059695 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.055074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:19.059695 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:19.055326 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:19.059695 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.055074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:19.059695 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:19.055530 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:19.060014 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.059767 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" event={"ID":"b347becc-88b4-42a1-a9bb-71a60030655f","Type":"ContainerStarted","Data":"14a153c9bd763bfb4adefa7c4073c679e4c8de5b01a86b511e6a5461b918b662"} Sep 17 20:49:19.065179 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.065103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgqm7" event={"ID":"fc6dd067-c1f7-44ad-9ec7-62bf9738c142","Type":"ContainerStarted","Data":"3d0f6d88061194a0055c1dd873f3b6de2457aeef271b088fca54f21435b4a366"} Sep 17 20:49:19.069847 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.069759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerStarted","Data":"9c2ead063dbed11662f804402164d9752f90608531395f1718decfa26e297a39"} Sep 17 20:49:19.077286 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.077190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4kl45" event={"ID":"077c8c39-4a85-4df3-a960-2ffd249b40f9","Type":"ContainerStarted","Data":"fdeb3ffd54453ba602ceefd65f9dde1253debf5cc55589b3eb9d9cc5b24910d5"} Sep 17 20:49:19.083744 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.083714 2579 generic.go:356] "Generic (PLEG): container finished" podID="868967e77ed612cb8308ea9845864a59" containerID="3066fdb714921f57b7962d221d412d375acf945abc20ab46a089f2f09c510886" exitCode=0 Sep 17 20:49:19.083849 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.083793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" event={"ID":"868967e77ed612cb8308ea9845864a59","Type":"ContainerDied","Data":"3066fdb714921f57b7962d221d412d375acf945abc20ab46a089f2f09c510886"} Sep 17 20:49:19.092784 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.092747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rtwxb" event={"ID":"81833eee-01cf-420e-9cfb-8b6705c2db18","Type":"ContainerStarted","Data":"4c475fcec9e4df2fb3bf01bf65dc9e60c72d968f41dda28416bbd57f2ca0315b"} Sep 17 20:49:19.095811 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.095782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8vvzl" event={"ID":"387082f5-712b-4e1c-820e-66c7f71a0250","Type":"ContainerStarted","Data":"4c22953967f8cb1cf8ca446bf9bf8abdd01ff5dd395dd5eedce8169af9de6e4b"} Sep 17 20:49:19.107018 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.106939 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phgq5" event={"ID":"9f02e492-cf56-4deb-8665-67180c2517cf","Type":"ContainerStarted","Data":"65d9550baf6831b0337376b718853ad0e807f1b91b5e679ca4cdabf58160d127"} Sep 17 20:49:19.110850 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.110821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"daaa270aab1bcfb7917737190ff8b6cf4092114722d3af52463fec7499c1c4d2"} Sep 17 20:49:19.121926 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.121897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gb8f6" event={"ID":"d769f3fe-96c8-4c71-9890-6d16a4869ec1","Type":"ContainerStarted","Data":"cd38fe3ebc9236207c5f75645b4152c7f10a1eebda53f015495a5dfb3872fca5"} Sep 17 20:49:19.137461 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:19.137419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" event={"ID":"576eb62884aeb095e963216a64f67066","Type":"ContainerStarted","Data":"06e6bdafe406f458eecfa372321d2e070c2fbc81436d6b6f39d46f563c350b85"} Sep 17 20:49:20.047866 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:20.047827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:20.048487 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.047974 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:20.150562 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:20.150523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" event={"ID":"868967e77ed612cb8308ea9845864a59","Type":"ContainerStarted","Data":"7ff3a6c1f981a6f4c9446e547ff585d0f235a2f3d337953a91a75825b794f0d4"} Sep 17 20:49:20.174766 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:20.174699 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-118.ec2.internal" podStartSLOduration=3.174677982 podStartE2EDuration="3.174677982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:19.153822106 +0000 UTC m=+4.628504455" watchObservedRunningTime="2026-09-17 20:49:20.174677982 +0000 UTC m=+5.649360333" Sep 17 20:49:20.587853 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:20.587816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:20.588033 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.587977 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:20.588109 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.588043 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.588022871 +0000 UTC m=+10.062705202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:20.688967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:20.688926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:20.689142 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:20.689001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:20.689207 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.689140 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:20.689271 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.689208 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.689188248 +0000 UTC m=+10.163870582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:20.689342 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.689299 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:20.689342 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.689313 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:20.689342 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.689327 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:20.689528 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:20.689365 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:24.689353488 +0000 UTC m=+10.164035814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:21.048042 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:21.047923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:21.048540 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:21.048058 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:21.048540 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:21.048475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:21.048681 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:21.048567 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:22.048298 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:22.048083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:22.048825 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:22.048346 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:23.048512 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:23.048477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:23.048965 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:23.048617 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:23.049521 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:23.048442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:23.049660 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:23.049636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:24.048048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:24.047792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:24.048261 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.048161 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:24.628275 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:24.628224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:24.628761 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.628422 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:24.628761 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.628500 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:49:32.628479627 +0000 UTC m=+18.103161972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:24.729444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:24.729411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:24.729647 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:24.729484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:24.729647 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.729604 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:24.729647 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.729618 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:24.729647 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.729640 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:24.729859 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.729655 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:24.729859 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.729668 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:32.72964751 +0000 UTC m=+18.204330061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:24.729859 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:24.729708 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:32.729693471 +0000 UTC m=+18.204375818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:25.048853 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:25.048767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:25.049026 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:25.048895 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:25.049092 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:25.049027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:25.049286 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:25.049251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:26.047536 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:26.047507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:26.047998 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:26.047647 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:27.047569 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:27.047537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:27.047936 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:27.047671 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:27.048150 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:27.048128 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:27.048274 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:27.048241 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:28.048076 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:28.048033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:28.048543 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:28.048170 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:29.047447 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:29.047417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:29.047598 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:29.047514 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:29.047598 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:29.047569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:29.047664 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:29.047648 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:30.048468 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:30.048442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:30.048826 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:30.048561 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:31.048005 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:31.047968 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:31.048176 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:31.047984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:31.048176 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:31.048088 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:31.048294 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:31.048193 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:32.048087 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:32.048057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:32.048546 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.048164 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:32.687946 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:32.687911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:32.688213 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.688037 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:32.688213 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.688092 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:49:48.688072204 +0000 UTC m=+34.162754546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:32.788445 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:32.788411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:32.788607 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:32.788454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:32.788607 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.788567 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:32.788607 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.788575 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:32.788607 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.788596 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:32.788607 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.788608 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:32.788866 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.788628 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:49:48.788609825 +0000 UTC m=+34.263292164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:32.788866 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:32.788646 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:48.788635793 +0000 UTC m=+34.263318116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:33.048315 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:33.048235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:33.048755 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:33.048361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:33.048755 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:33.048402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:33.048755 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:33.048510 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:34.047504 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:34.047476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:34.047614 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:34.047587 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:35.050102 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:35.050075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:35.050102 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:35.050088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:35.050561 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:35.050159 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:35.050561 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:35.050291 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:36.048307 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:36.048269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:36.048494 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:36.048401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:37.048411 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:37.048333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:37.048865 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:37.048482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:37.048865 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:37.048540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:37.048865 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:37.048647 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:38.047732 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.047579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:38.047878 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:38.047806 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:38.192832 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.192760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rtwxb" event={"ID":"81833eee-01cf-420e-9cfb-8b6705c2db18","Type":"ContainerStarted","Data":"97d5c95c41c50ac13701d26092e64491a2171e586de90acd3eee3bc4ce75fa88"} Sep 17 20:49:38.194571 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.194546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8vvzl" event={"ID":"387082f5-712b-4e1c-820e-66c7f71a0250","Type":"ContainerStarted","Data":"7b0b5994cd25e838992fb3200a02a489ebfebb129f0d8c49fe2f9fff000e64f5"} Sep 17 20:49:38.196186 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.196164 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phgq5" event={"ID":"9f02e492-cf56-4deb-8665-67180c2517cf","Type":"ContainerStarted","Data":"9defdbf012bf28f03ee993b69282d4cb37819f86099f8ef0acc47140992263be"} Sep 17 20:49:38.199564 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.199540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"1df4420a8d7c850dc74259da8264bcf52e1ca6ec3efb00e49073707f4bf18678"} Sep 17 20:49:38.199658 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.199615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"8698af776fec72ac9fcaabc0fab9b6dd130f9a3aae8a131c4023fe5fc06d6c65"} Sep 17 20:49:38.199705 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.199663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"3c7463c610112c8dbb40f5df4ddb0f826957abb41627ec44e2a06cb4231fe6a5"} Sep 17 20:49:38.199752 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.199709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"5890ccbe7d07824cd087e1adfe970bd28824d6b5e505a5cccbf2fd3e55e0889d"} Sep 17 20:49:38.199796 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.199757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"8379afd1467a68200a1236117b95da6a5b63ead0dc11da7bdd73e3c8523f9a59"} Sep 17 20:49:38.199829 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.199805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"52365efd9052f4587dda6eb2bc49ac1f43de78593db10abaf93a587af8777990"} Sep 17 20:49:38.201100 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.201071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gb8f6" event={"ID":"d769f3fe-96c8-4c71-9890-6d16a4869ec1","Type":"ContainerStarted","Data":"0e33faa63686f2424e04c647e67bb8b683a94bc7313b8c99c9bef0e6275ad20d"} Sep 17 20:49:38.202705 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.202674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" event={"ID":"b347becc-88b4-42a1-a9bb-71a60030655f","Type":"ContainerStarted","Data":"b3523138582d78ad59b3b99c64859257e78fd8af58793e52ab783afa5dfbf73f"} Sep 17 20:49:38.204245 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.204221 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgqm7" event={"ID":"fc6dd067-c1f7-44ad-9ec7-62bf9738c142","Type":"ContainerStarted","Data":"df80162a49981e0ff4cabbc1b547f3418c076a6a99c150a24907dfcf87f9ac6f"} Sep 17 20:49:38.206001 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.205978 2579 generic.go:356] "Generic (PLEG): container finished" podID="e434119f-e83f-428a-9061-2e131afb5f4f" containerID="d1e26e2e5ec06cd9ae3601ad4997101d21f122c5fb4964d15ec72315f926c2c8" exitCode=0 Sep 17 20:49:38.206087 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.206012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerDied","Data":"d1e26e2e5ec06cd9ae3601ad4997101d21f122c5fb4964d15ec72315f926c2c8"} Sep 17 20:49:38.212045 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.212008 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-118.ec2.internal" podStartSLOduration=21.211995102 podStartE2EDuration="21.211995102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:20.175014013 +0000 UTC m=+5.649696362" watchObservedRunningTime="2026-09-17 20:49:38.211995102 +0000 UTC m=+23.686677445" Sep 17 20:49:38.234976 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.234930 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rtwxb" podStartSLOduration=3.102481772 podStartE2EDuration="22.234915156s" totalImagesPullingTime="19.132433384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:38.213078196 +0000 UTC m=+23.687760540" watchObservedRunningTime="2026-09-17 20:49:38.234915156 +0000 UTC m=+23.709597516" Sep 17 20:49:38.235264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.235238 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pgqm7" podStartSLOduration=3.158232659 podStartE2EDuration="22.235232384s" totalImagesPullingTime="19.076999725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:38.234820478 +0000 UTC m=+23.709502823" watchObservedRunningTime="2026-09-17 20:49:38.235232384 +0000 UTC m=+23.709914727" Sep 17 20:49:38.252895 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.252860 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-phgq5" podStartSLOduration=4.13336573 podStartE2EDuration="23.252849256s" totalImagesPullingTime="19.119483526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:38.252650758 +0000 UTC m=+23.727333104" watchObservedRunningTime="2026-09-17 20:49:38.252849256 +0000 UTC m=+23.727531599" Sep 17 20:49:38.269336 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.269289 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gb8f6" podStartSLOduration=4.191155143 podStartE2EDuration="23.269278401s" totalImagesPullingTime="19.078123258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:38.269123011 +0000 UTC m=+23.743805357" watchObservedRunningTime="2026-09-17 20:49:38.269278401 +0000 UTC m=+23.743960744" Sep 17 20:49:38.298748 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:38.298706 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8vvzl" podStartSLOduration=3.232936427 podStartE2EDuration="22.298689512s" totalImagesPullingTime="19.065753085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:38.298198958 +0000 UTC m=+23.772881303" watchObservedRunningTime="2026-09-17 20:49:38.298689512 +0000 UTC m=+23.773371855" Sep 17 20:49:39.021220 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.021052 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 20:49:39.048181 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.048158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:39.048310 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.048184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:39.048310 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:39.048290 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:39.048459 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:39.048435 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:39.210506 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.210404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" event={"ID":"b347becc-88b4-42a1-a9bb-71a60030655f","Type":"ContainerStarted","Data":"f23d8b347f9a3e9174b77ead60626e8901caf7543560422be5a93cb6850c2b6e"} Sep 17 20:49:39.212022 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.211992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4kl45" event={"ID":"077c8c39-4a85-4df3-a960-2ffd249b40f9","Type":"ContainerStarted","Data":"5dd1248e5f11987df3331f095c051829d84091f91761c870aabaa0090af0fd98"} Sep 17 20:49:39.230375 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.230323 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4kl45" podStartSLOduration=4.147489914 podStartE2EDuration="23.230304868s" totalImagesPullingTime="19.082814954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:39.229997007 +0000 UTC m=+24.704679363" watchObservedRunningTime="2026-09-17 20:49:39.230304868 +0000 UTC m=+24.704987213" Sep 17 20:49:39.510800 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.510732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:39.511416 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:39.511392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:40.012966 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:40.012881 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:49:39.021211283Z","UUID":"c3f91d45-96ad-4fb6-b02b-aaae0483084b","Handler":null,"Name":"","Endpoint":""} Sep 17 20:49:40.014566 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:40.014547 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 20:49:40.014566 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:40.014579 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 20:49:40.048420 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:40.048395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:40.048551 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:40.048529 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:40.218182 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:40.218103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"f874e27327ba5b5d51de5bd863daf3292023e735a3c6971fc4cff1c2cebdb6cc"} Sep 17 20:49:41.047992 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:41.047958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:41.048200 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:41.047958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:41.048200 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:41.048074 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:41.048200 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:41.048171 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:41.222218 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:41.222175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" event={"ID":"b347becc-88b4-42a1-a9bb-71a60030655f","Type":"ContainerStarted","Data":"4536f587d80d3fe4a2d3f459c433bfa628f93e82dd0ae2b19e5dcdb1f960ef12"} Sep 17 20:49:41.222218 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:41.222211 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 20:49:42.048230 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:42.048190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:42.048423 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:42.048308 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:43.048291 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.048054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:43.048806 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.048117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:43.048806 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:43.048410 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:43.048806 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:43.048453 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:43.228424 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.228392 2579 generic.go:356] "Generic (PLEG): container finished" podID="e434119f-e83f-428a-9061-2e131afb5f4f" containerID="6c9d55b87ba65f46eac4bf79430c12590111f887e6f8c1645b99782412ff2f98" exitCode=0 Sep 17 20:49:43.228571 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.228473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerDied","Data":"6c9d55b87ba65f46eac4bf79430c12590111f887e6f8c1645b99782412ff2f98"} Sep 17 20:49:43.233325 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.233299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" event={"ID":"f662058f-fd45-4280-85de-0547407d6805","Type":"ContainerStarted","Data":"e8156a80098d01215c6201ca75c07f9679d72606ca49426077ae0a30a33597a9"} Sep 17 20:49:43.233615 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.233590 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:43.233712 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.233670 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:43.233777 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.233718 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:43.250846 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.250824 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:43.251092 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.251076 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:49:43.252074 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.252029 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgzkd" podStartSLOduration=6.553087249 podStartE2EDuration="28.252016039s" totalImagesPullingTime="21.69892879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:41.247774409 +0000 UTC m=+26.722456754" watchObservedRunningTime="2026-09-17 20:49:43.252016039 +0000 UTC m=+28.726698383" Sep 17 20:49:43.280801 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.280766 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" podStartSLOduration=8.088231211 podStartE2EDuration="27.280754727s" totalImagesPullingTime="19.192523516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:43.278944186 +0000 UTC m=+28.753626530" watchObservedRunningTime="2026-09-17 20:49:43.280754727 +0000 UTC m=+28.755437070" Sep 17 20:49:43.727402 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.727305 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:43.727545 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.727482 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 20:49:43.728222 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:43.728191 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gb8f6" Sep 17 20:49:44.048349 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.048294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:44.048898 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:44.048425 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:44.239175 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.239140 2579 generic.go:356] "Generic (PLEG): container finished" podID="e434119f-e83f-428a-9061-2e131afb5f4f" containerID="179934fffdd78762b0d5d9c895f7104a52e5f80337792c0ed40de8844a3fc4a8" exitCode=0 Sep 17 20:49:44.239328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.239228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerDied","Data":"179934fffdd78762b0d5d9c895f7104a52e5f80337792c0ed40de8844a3fc4a8"} Sep 17 20:49:44.482337 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.482272 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqdcz"] Sep 17 20:49:44.482486 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.482408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:44.482541 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:44.482481 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:44.485985 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.485966 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z556m"] Sep 17 20:49:44.486108 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.486044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:44.486145 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:44.486105 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:44.488217 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.488191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f8sff"] Sep 17 20:49:44.488310 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:44.488275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:44.488408 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:44.488390 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:45.243530 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:45.243267 2579 generic.go:356] "Generic (PLEG): container finished" podID="e434119f-e83f-428a-9061-2e131afb5f4f" containerID="101a91aea4c2bb03fb64f3872f24409fa9b525af37cadc44daace2d03cc4ba86" exitCode=0 Sep 17 20:49:45.243868 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:45.243343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerDied","Data":"101a91aea4c2bb03fb64f3872f24409fa9b525af37cadc44daace2d03cc4ba86"} Sep 17 20:49:46.047906 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:46.047872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:46.047906 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:46.047911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:46.048150 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:46.047992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:46.048150 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:46.047995 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:46.048150 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:46.048121 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:46.048301 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:46.048213 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:48.048320 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:48.048244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:48.048320 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:48.048294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:48.048913 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:48.048239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:48.048913 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.048395 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:48.048913 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.048479 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:48.048913 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.048574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:48.712312 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:48.712273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:48.712531 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.712428 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:48.712531 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.712511 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:50:20.712489917 +0000 UTC m=+66.187172257 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 20:49:48.812818 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:48.812785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:48.812984 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:48.812827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:48.812984 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.812927 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:48.812984 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.812947 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 20:49:48.812984 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.812970 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 20:49:48.812984 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.812978 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8w556 for pod openshift-network-diagnostics/network-check-target-z556m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:48.812984 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.812987 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret podName:9bca1fc5-af86-4106-b5f5-b7ffdefab53a nodeName:}" failed. No retries permitted until 2026-09-17 20:50:20.81297004 +0000 UTC m=+66.287652384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret") pod "global-pull-secret-syncer-xqdcz" (UID: "9bca1fc5-af86-4106-b5f5-b7ffdefab53a") : object "kube-system"/"original-pull-secret" not registered Sep 17 20:49:48.813302 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:48.813020 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556 podName:90b744c7-94c6-400f-91eb-def724247bc0 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:20.813006378 +0000 UTC m=+66.287688700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8w556" (UniqueName: "kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556") pod "network-check-target-z556m" (UID: "90b744c7-94c6-400f-91eb-def724247bc0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 20:49:50.047456 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.047437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:50.047784 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.047469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:50.047784 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.047523 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z556m" podUID="90b744c7-94c6-400f-91eb-def724247bc0" Sep 17 20:49:50.047784 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.047557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:50.047784 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.047631 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqdcz" podUID="9bca1fc5-af86-4106-b5f5-b7ffdefab53a" Sep 17 20:49:50.047784 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.047712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8sff" podUID="e85d8f30-029c-4596-a39d-101227da18aa" Sep 17 20:49:50.260208 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.260175 2579 generic.go:356] "Generic (PLEG): container finished" podID="e434119f-e83f-428a-9061-2e131afb5f4f" containerID="f975a3217c238e2e5233b9149d7bce85d97f28c3979a5b6f9d03bc518950667a" exitCode=0 Sep 17 20:49:50.260328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.260215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerDied","Data":"f975a3217c238e2e5233b9149d7bce85d97f28c3979a5b6f9d03bc518950667a"} Sep 17 20:49:50.351512 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.351490 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-118.ec2.internal" event="NodeReady" Sep 17 20:49:50.351602 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.351580 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 20:49:50.390441 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.390417 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b6c8df995-pbvm8"] Sep 17 20:49:50.414356 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.414332 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nt4pt"] Sep 17 20:49:50.414486 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.414448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.420882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.420244 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 20:49:50.420882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.420307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 20:49:50.420882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.420429 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 20:49:50.420882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.420517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p2pj8\"" Sep 17 20:49:50.427297 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.427266 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 20:49:50.436563 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.436531 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8jqfh"] Sep 17 20:49:50.436655 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.436642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.439649 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.439631 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxtkj\"" Sep 17 20:49:50.439649 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.439639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 20:49:50.439798 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.439656 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 20:49:50.439798 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.439661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 20:49:50.458793 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.458749 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b6c8df995-pbvm8"] Sep 17 20:49:50.458793 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.458776 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nt4pt"] Sep 17 20:49:50.458793 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.458789 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jqfh"] Sep 17 20:49:50.458922 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.458858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.461443 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.461429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 20:49:50.461515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.461453 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 20:49:50.461550 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.461459 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6qzc\"" Sep 17 20:49:50.524645 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqmn\" (UniqueName: \"kubernetes.io/projected/d7d3912c-f311-4cda-b935-cf83db10c7c4-kube-api-access-ctqmn\") pod \"ingress-canary-nt4pt\" (UID: \"d7d3912c-f311-4cda-b935-cf83db10c7c4\") " pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.524743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-image-registry-private-configuration\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-registry-certificates\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5tf\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-kube-api-access-6z5tf\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7d3912c-f311-4cda-b935-cf83db10c7c4-cert\") pod \"ingress-canary-nt4pt\" (UID: \"d7d3912c-f311-4cda-b935-cf83db10c7c4\") " pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.524912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-bound-sa-token\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d974800-a495-41e7-87e6-69e28de6156f-ca-trust-extracted\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-trusted-ca\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.524912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.524834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-installation-pull-secrets\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625435 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqmn\" (UniqueName: \"kubernetes.io/projected/d7d3912c-f311-4cda-b935-cf83db10c7c4-kube-api-access-ctqmn\") pod \"ingress-canary-nt4pt\" (UID: \"d7d3912c-f311-4cda-b935-cf83db10c7c4\") " pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.625517 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-image-registry-private-configuration\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625517 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-registry-certificates\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625517 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5tf\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-kube-api-access-6z5tf\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625517 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.625722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74fbf86b-d686-4752-8c33-1607b699dece-tmp-dir\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.625722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7d3912c-f311-4cda-b935-cf83db10c7c4-cert\") pod \"ingress-canary-nt4pt\" (UID: \"d7d3912c-f311-4cda-b935-cf83db10c7c4\") " pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.625722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-bound-sa-token\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d974800-a495-41e7-87e6-69e28de6156f-ca-trust-extracted\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-trusted-ca\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckv8\" (UniqueName: \"kubernetes.io/projected/74fbf86b-d686-4752-8c33-1607b699dece-kube-api-access-wckv8\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.625942 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.625942 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.625768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-installation-pull-secrets\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.626474 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.626076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74fbf86b-d686-4752-8c33-1607b699dece-config-volume\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.626474 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.626232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d974800-a495-41e7-87e6-69e28de6156f-ca-trust-extracted\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.626474 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.626243 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:49:50.626474 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.626401 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:49:50.626474 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.626467 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:49:51.126438696 +0000 UTC m=+36.601121018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:49:50.626781 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.626669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-registry-certificates\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.627234 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.627072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-trusted-ca\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.631548 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.631519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-installation-pull-secrets\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.631760 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.631738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-image-registry-private-configuration\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.632855 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.632829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7d3912c-f311-4cda-b935-cf83db10c7c4-cert\") pod \"ingress-canary-nt4pt\" (UID: \"d7d3912c-f311-4cda-b935-cf83db10c7c4\") " pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.634058 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.634031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqmn\" (UniqueName: \"kubernetes.io/projected/d7d3912c-f311-4cda-b935-cf83db10c7c4-kube-api-access-ctqmn\") pod \"ingress-canary-nt4pt\" (UID: \"d7d3912c-f311-4cda-b935-cf83db10c7c4\") " pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:50.634164 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.634147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5tf\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-kube-api-access-6z5tf\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.634804 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.634773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-bound-sa-token\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:50.727048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.726997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.727048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.727024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74fbf86b-d686-4752-8c33-1607b699dece-tmp-dir\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.727048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.727044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckv8\" (UniqueName: \"kubernetes.io/projected/74fbf86b-d686-4752-8c33-1607b699dece-kube-api-access-wckv8\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.727200 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.727084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74fbf86b-d686-4752-8c33-1607b699dece-config-volume\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.727200 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.727122 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:49:50.727200 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:50.727170 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:49:51.227156469 +0000 UTC m=+36.701838792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:49:50.727419 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.727397 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74fbf86b-d686-4752-8c33-1607b699dece-tmp-dir\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.727592 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.727578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74fbf86b-d686-4752-8c33-1607b699dece-config-volume\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.737924 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.737902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckv8\" (UniqueName: \"kubernetes.io/projected/74fbf86b-d686-4752-8c33-1607b699dece-kube-api-access-wckv8\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:50.797913 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:50.797894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nt4pt" Sep 17 20:49:51.123609 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:51.123587 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nt4pt"] Sep 17 20:49:51.125708 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:49:51.125683 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d3912c_f311_4cda_b935_cf83db10c7c4.slice/crio-ae743bf39c6b8c5a35d37602b381c1cb23d2399513330f2000b573647dd3165d WatchSource:0}: Error finding container ae743bf39c6b8c5a35d37602b381c1cb23d2399513330f2000b573647dd3165d: Status 404 returned error can't find the container with id ae743bf39c6b8c5a35d37602b381c1cb23d2399513330f2000b573647dd3165d Sep 17 20:49:51.131261 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:51.131236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:51.131403 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:51.131386 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:49:51.131450 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:51.131405 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:49:51.131486 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:51.131455 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:49:52.131437518 +0000 UTC m=+37.606119855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:49:51.231962 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:51.231943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:51.232089 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:51.232073 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:49:51.232132 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:51.232123 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:49:52.232111255 +0000 UTC m=+37.706793577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:49:51.263993 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:51.263966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nt4pt" event={"ID":"d7d3912c-f311-4cda-b935-cf83db10c7c4","Type":"ContainerStarted","Data":"ae743bf39c6b8c5a35d37602b381c1cb23d2399513330f2000b573647dd3165d"} Sep 17 20:49:51.267059 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:51.267038 2579 generic.go:356] "Generic (PLEG): container finished" podID="e434119f-e83f-428a-9061-2e131afb5f4f" containerID="ff032100bc9d93748225fb17495fa5045ffd66de6c1dec027c4b08430999d2ec" exitCode=0 Sep 17 20:49:51.267137 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:51.267075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerDied","Data":"ff032100bc9d93748225fb17495fa5045ffd66de6c1dec027c4b08430999d2ec"} Sep 17 20:49:52.048095 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.048058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:49:52.048294 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.048068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:49:52.048294 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.048203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:49:52.051350 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.051326 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 20:49:52.052706 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.052686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-775ms\"" Sep 17 20:49:52.052801 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.052752 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j729c\"" Sep 17 20:49:52.052801 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.052686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 20:49:52.052933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.052686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 20:49:52.052933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.052826 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 20:49:52.138138 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.138116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:52.138612 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:52.138227 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:49:52.138612 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:52.138247 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:49:52.138612 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:52.138315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.138293972 +0000 UTC m=+39.612976302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:49:52.239059 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.239028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:52.239211 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:52.239192 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:49:52.239259 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:52.239252 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.239237679 +0000 UTC m=+39.713920001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:49:52.274018 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.273990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t72xc" event={"ID":"e434119f-e83f-428a-9061-2e131afb5f4f","Type":"ContainerStarted","Data":"7a145f4456b9cd5d24468ebe937b2a7e3c02b3bf8eca9c934063d4f0c1e7aafc"} Sep 17 20:49:52.297887 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:52.297849 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t72xc" podStartSLOduration=7.100926047 podStartE2EDuration="36.297835804s" totalImagesPullingTime="29.196909757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:52.296055738 +0000 UTC m=+37.770738081" watchObservedRunningTime="2026-09-17 20:49:52.297835804 +0000 UTC m=+37.772518148" Sep 17 20:49:54.152847 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:54.152662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:54.153141 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:54.152812 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:49:54.153141 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:54.152866 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:49:54.153141 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:54.152915 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:49:58.152901815 +0000 UTC m=+43.627584151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:49:54.254047 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:54.254017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:54.254138 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:54.254125 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:49:54.254186 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:54.254177 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:49:58.254164115 +0000 UTC m=+43.728846436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:49:54.280272 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:54.280242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nt4pt" event={"ID":"d7d3912c-f311-4cda-b935-cf83db10c7c4","Type":"ContainerStarted","Data":"bdbf76964e5c31ac5e0738c3881d708e0c230da7967475271c3840ea850c426b"} Sep 17 20:49:54.299116 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:54.299053 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nt4pt" podStartSLOduration=1.895284965 podStartE2EDuration="4.299041178s" totalImagesPullingTime="2.403756213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:54.298834822 +0000 UTC m=+39.773517167" watchObservedRunningTime="2026-09-17 20:49:54.299041178 +0000 UTC m=+39.773723513" Sep 17 20:49:58.181214 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:58.181177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:49:58.181748 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:58.181323 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:49:58.181748 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:58.181336 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:49:58.181748 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:58.181425 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:50:06.181409113 +0000 UTC m=+51.656091454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:49:58.282118 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:49:58.282086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:49:58.282236 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:58.282209 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:49:58.282292 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:49:58.282257 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:50:06.282244892 +0000 UTC m=+51.756927215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:50:06.236114 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:06.236075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:50:06.236622 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:06.236192 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:50:06.236622 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:06.236204 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:50:06.236622 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:06.236267 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:50:22.236251313 +0000 UTC m=+67.710933636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:50:06.336833 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:06.336799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:50:06.336982 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:06.336912 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:50:06.336982 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:06.336969 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:50:22.33695559 +0000 UTC m=+67.811637913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:50:07.126696 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.126667 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-shcng"] Sep 17 20:50:07.128631 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.128613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.130474 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.130452 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm"] Sep 17 20:50:07.132246 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.132212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.132361 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.132254 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 20:50:07.132785 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.132763 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 20:50:07.132785 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.132782 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 20:50:07.132929 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.132765 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 20:50:07.132929 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.132804 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-b5xtw\"" Sep 17 20:50:07.137107 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.137085 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 20:50:07.137182 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.137087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 17 20:50:07.138444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.138424 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 17 20:50:07.138542 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.138505 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lrzjv\"" Sep 17 20:50:07.140065 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.140047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 20:50:07.140738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.140722 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 20:50:07.143009 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.142980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-serving-cert\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.143009 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fptj\" (UniqueName: \"kubernetes.io/projected/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-kube-api-access-5fptj\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.143130 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-tmp\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.143130 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-snapshots\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.143130 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.143130 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.143248 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2h5\" (UniqueName: \"kubernetes.io/projected/2f0c9995-a015-4054-8ba6-0b39ae783f5e-kube-api-access-pr2h5\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.143248 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.143248 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.143218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f0c9995-a015-4054-8ba6-0b39ae783f5e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.145568 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.145538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm"] Sep 17 20:50:07.149150 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.149127 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-shcng"] Sep 17 20:50:07.243932 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.243908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.243955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f0c9995-a015-4054-8ba6-0b39ae783f5e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-serving-cert\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.244042 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fptj\" (UniqueName: \"kubernetes.io/projected/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-kube-api-access-5fptj\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.244108 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls podName:2f0c9995-a015-4054-8ba6-0b39ae783f5e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:07.744090541 +0000 UTC m=+53.218772882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h9drm" (UID: "2f0c9995-a015-4054-8ba6-0b39ae783f5e") : secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-tmp\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-snapshots\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244677 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2h5\" (UniqueName: \"kubernetes.io/projected/2f0c9995-a015-4054-8ba6-0b39ae783f5e-kube-api-access-pr2h5\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.244677 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-tmp\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244808 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244888 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f0c9995-a015-4054-8ba6-0b39ae783f5e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.244888 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-snapshots\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.244987 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.244969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.247668 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.247652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-serving-cert\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.256104 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.256076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2h5\" (UniqueName: \"kubernetes.io/projected/2f0c9995-a015-4054-8ba6-0b39ae783f5e-kube-api-access-pr2h5\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.256614 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.256594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fptj\" (UniqueName: \"kubernetes.io/projected/8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9-kube-api-access-5fptj\") pod \"insights-operator-6f9c644dd9-shcng\" (UID: \"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9\") " pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.288876 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.288858 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd"] Sep 17 20:50:07.291584 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.291569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.295295 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.295281 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 20:50:07.295634 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.295619 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 20:50:07.295717 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.295639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gnb5q\"" Sep 17 20:50:07.295717 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.295665 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 20:50:07.315314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.315294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd"] Sep 17 20:50:07.345677 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.345654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.345758 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.345695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsl8x\" (UniqueName: \"kubernetes.io/projected/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-kube-api-access-zsl8x\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.440237 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.440181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" Sep 17 20:50:07.446611 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.446590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.446672 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.446655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsl8x\" (UniqueName: \"kubernetes.io/projected/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-kube-api-access-zsl8x\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.446931 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.446886 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 20:50:07.447004 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.446991 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls podName:89e0867e-07e3-4beb-97b3-6fabfe44cfbb nodeName:}" failed. No retries permitted until 2026-09-17 20:50:07.946971001 +0000 UTC m=+53.421653326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gs2qd" (UID: "89e0867e-07e3-4beb-97b3-6fabfe44cfbb") : secret "samples-operator-tls" not found Sep 17 20:50:07.457703 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.457679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsl8x\" (UniqueName: \"kubernetes.io/projected/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-kube-api-access-zsl8x\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.708805 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.708739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-shcng"] Sep 17 20:50:07.711506 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:07.711482 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbe1c9e_0f9e_44f2_b3d0_530393ef4ed9.slice/crio-eccb968fb73ef8c1654acfe425850a1d44a32c80ba3387c8dbe5b5eb8a5e25df WatchSource:0}: Error finding container eccb968fb73ef8c1654acfe425850a1d44a32c80ba3387c8dbe5b5eb8a5e25df: Status 404 returned error can't find the container with id eccb968fb73ef8c1654acfe425850a1d44a32c80ba3387c8dbe5b5eb8a5e25df Sep 17 20:50:07.748986 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.748957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:07.749124 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.749105 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:07.749176 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.749166 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls podName:2f0c9995-a015-4054-8ba6-0b39ae783f5e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:08.74915155 +0000 UTC m=+54.223833903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h9drm" (UID: "2f0c9995-a015-4054-8ba6-0b39ae783f5e") : secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:07.950439 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:07.950414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:07.950579 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.950561 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 20:50:07.950626 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:07.950621 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls podName:89e0867e-07e3-4beb-97b3-6fabfe44cfbb nodeName:}" failed. No retries permitted until 2026-09-17 20:50:08.950606641 +0000 UTC m=+54.425288994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gs2qd" (UID: "89e0867e-07e3-4beb-97b3-6fabfe44cfbb") : secret "samples-operator-tls" not found Sep 17 20:50:08.317847 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:08.317810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" event={"ID":"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9","Type":"ContainerStarted","Data":"eccb968fb73ef8c1654acfe425850a1d44a32c80ba3387c8dbe5b5eb8a5e25df"} Sep 17 20:50:08.757008 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:08.756924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:08.757166 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:08.757075 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:08.757166 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:08.757159 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls podName:2f0c9995-a015-4054-8ba6-0b39ae783f5e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:10.757137874 +0000 UTC m=+56.231820202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h9drm" (UID: "2f0c9995-a015-4054-8ba6-0b39ae783f5e") : secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:08.958417 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:08.958380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:08.958575 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:08.958481 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 20:50:08.958575 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:08.958559 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls podName:89e0867e-07e3-4beb-97b3-6fabfe44cfbb nodeName:}" failed. No retries permitted until 2026-09-17 20:50:10.958538066 +0000 UTC m=+56.433220394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gs2qd" (UID: "89e0867e-07e3-4beb-97b3-6fabfe44cfbb") : secret "samples-operator-tls" not found Sep 17 20:50:10.324633 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:10.324601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" event={"ID":"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9","Type":"ContainerStarted","Data":"8f6beddac2297c969d8000d460e303bcef2c659a891cdce02b80288804699998"} Sep 17 20:50:10.343554 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:10.343510 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" podStartSLOduration=1.397107354 podStartE2EDuration="3.343493347s" totalImagesPullingTime="1.946385993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:10.342194924 +0000 UTC m=+55.816877271" watchObservedRunningTime="2026-09-17 20:50:10.343493347 +0000 UTC m=+55.818175688" Sep 17 20:50:10.772092 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:10.772064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:10.772228 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:10.772174 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:10.772282 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:10.772235 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls podName:2f0c9995-a015-4054-8ba6-0b39ae783f5e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:14.772213395 +0000 UTC m=+60.246895717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h9drm" (UID: "2f0c9995-a015-4054-8ba6-0b39ae783f5e") : secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:10.973602 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:10.973570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:10.973745 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:10.973722 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 20:50:10.973802 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:10.973785 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls podName:89e0867e-07e3-4beb-97b3-6fabfe44cfbb nodeName:}" failed. No retries permitted until 2026-09-17 20:50:14.973767623 +0000 UTC m=+60.448449949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gs2qd" (UID: "89e0867e-07e3-4beb-97b3-6fabfe44cfbb") : secret "samples-operator-tls" not found Sep 17 20:50:11.027515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.027453 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rsw4z"] Sep 17 20:50:11.030584 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.030561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.033311 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.033287 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 20:50:11.033609 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.033589 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h4vct\"" Sep 17 20:50:11.033720 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.033614 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 20:50:11.056643 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.056620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rsw4z"] Sep 17 20:50:11.074755 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.074732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.074853 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.074773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6918ad-3c4a-445e-932a-61c856a69da7-crio-socket\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.074853 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.074795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zp2\" (UniqueName: \"kubernetes.io/projected/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-api-access-26zp2\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.074928 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.074891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.074964 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.074931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6918ad-3c4a-445e-932a-61c856a69da7-data-volume\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175586 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.175562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175685 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.175604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6918ad-3c4a-445e-932a-61c856a69da7-crio-socket\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175685 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.175621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zp2\" (UniqueName: \"kubernetes.io/projected/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-api-access-26zp2\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175685 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.175672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175685 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.175674 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 20:50:11.175905 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.175683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b6918ad-3c4a-445e-932a-61c856a69da7-crio-socket\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175905 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.175709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6918ad-3c4a-445e-932a-61c856a69da7-data-volume\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.175905 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.175743 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.675722618 +0000 UTC m=+57.150404956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : configmap "kube-rbac-proxy" not found Sep 17 20:50:11.175905 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.175787 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 20:50:11.175905 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.175825 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:11.675811797 +0000 UTC m=+57.150494123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : secret "insights-runtime-extractor-tls" not found Sep 17 20:50:11.176097 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.176006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b6918ad-3c4a-445e-932a-61c856a69da7-data-volume\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.185818 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.185784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zp2\" (UniqueName: \"kubernetes.io/projected/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-api-access-26zp2\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.680740 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.680712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.681059 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:11.680797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:11.681059 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.680853 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 20:50:11.681059 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.680882 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 20:50:11.681059 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.680918 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:12.680900627 +0000 UTC m=+58.155582949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : secret "insights-runtime-extractor-tls" not found Sep 17 20:50:11.681059 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:11.680933 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:12.680926555 +0000 UTC m=+58.155608877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : configmap "kube-rbac-proxy" not found Sep 17 20:50:12.687654 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:12.687621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:12.688022 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:12.687685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:12.688022 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:12.687754 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 20:50:12.688022 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:12.687801 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 20:50:12.688022 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:12.687827 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:14.68781181 +0000 UTC m=+60.162494132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : configmap "kube-rbac-proxy" not found Sep 17 20:50:12.688022 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:12.687847 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:14.68783661 +0000 UTC m=+60.162518932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : secret "insights-runtime-extractor-tls" not found Sep 17 20:50:12.975257 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:12.975194 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgqm7_fc6dd067-c1f7-44ad-9ec7-62bf9738c142/dns-node-resolver/0.log" Sep 17 20:50:13.775532 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:13.775500 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8vvzl_387082f5-712b-4e1c-820e-66c7f71a0250/node-ca/0.log" Sep 17 20:50:14.703614 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:14.703563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:14.703812 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:14.703675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:14.703812 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:14.703715 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 20:50:14.703812 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:14.703803 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:18.703784575 +0000 UTC m=+64.178466897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : configmap "kube-rbac-proxy" not found Sep 17 20:50:14.704006 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:14.703863 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 20:50:14.704006 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:14.703950 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:18.703929206 +0000 UTC m=+64.178611542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : secret "insights-runtime-extractor-tls" not found Sep 17 20:50:14.804734 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:14.804695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:14.805092 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:14.804825 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:14.805092 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:14.804879 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls podName:2f0c9995-a015-4054-8ba6-0b39ae783f5e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:22.804866146 +0000 UTC m=+68.279548469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h9drm" (UID: "2f0c9995-a015-4054-8ba6-0b39ae783f5e") : secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:14.974670 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:14.974611 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nt4pt_d7d3912c-f311-4cda-b935-cf83db10c7c4/serve-healthcheck-canary/0.log" Sep 17 20:50:15.006403 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:15.006357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:15.006542 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:15.006465 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 20:50:15.006542 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:15.006511 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls podName:89e0867e-07e3-4beb-97b3-6fabfe44cfbb nodeName:}" failed. No retries permitted until 2026-09-17 20:50:23.006499317 +0000 UTC m=+68.481181640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gs2qd" (UID: "89e0867e-07e3-4beb-97b3-6fabfe44cfbb") : secret "samples-operator-tls" not found Sep 17 20:50:15.254183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:15.254117 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnbqp" Sep 17 20:50:18.735117 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:18.735081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:18.735565 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:18.735185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:18.735565 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:18.735240 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 20:50:18.735565 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:18.735265 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 20:50:18.735565 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:18.735309 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.735291666 +0000 UTC m=+72.209974011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : secret "insights-runtime-extractor-tls" not found Sep 17 20:50:18.735565 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:18.735326 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.735318944 +0000 UTC m=+72.210001265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : configmap "kube-rbac-proxy" not found Sep 17 20:50:20.749147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.749115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:50:20.753114 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.753093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 20:50:20.759854 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:20.759839 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 20:50:20.759932 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:20.759890 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs podName:e85d8f30-029c-4596-a39d-101227da18aa nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.759875464 +0000 UTC m=+130.234557790 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs") pod "network-metrics-daemon-f8sff" (UID: "e85d8f30-029c-4596-a39d-101227da18aa") : secret "metrics-daemon-secret" not found Sep 17 20:50:20.849711 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.849680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:50:20.849819 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.849718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:50:20.854112 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.854090 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 20:50:20.854210 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.854154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 20:50:20.863175 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.863148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bca1fc5-af86-4106-b5f5-b7ffdefab53a-original-pull-secret\") pod \"global-pull-secret-syncer-xqdcz\" (UID: \"9bca1fc5-af86-4106-b5f5-b7ffdefab53a\") " pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:50:20.863597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.863580 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 20:50:20.872055 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.872034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqdcz" Sep 17 20:50:20.873784 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:20.873765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w556\" (UniqueName: \"kubernetes.io/projected/90b744c7-94c6-400f-91eb-def724247bc0-kube-api-access-8w556\") pod \"network-check-target-z556m\" (UID: \"90b744c7-94c6-400f-91eb-def724247bc0\") " pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:50:21.163557 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:21.163530 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j729c\"" Sep 17 20:50:21.167838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:21.167815 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqdcz"] Sep 17 20:50:21.170672 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:21.170651 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:50:21.170878 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:21.170840 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bca1fc5_af86_4106_b5f5_b7ffdefab53a.slice/crio-1925d0a83bd14663ec8dd133c9f5c63d4d858fe3bcf7e2ec8f8fa47dbb9f0471 WatchSource:0}: Error finding container 1925d0a83bd14663ec8dd133c9f5c63d4d858fe3bcf7e2ec8f8fa47dbb9f0471: Status 404 returned error can't find the container with id 1925d0a83bd14663ec8dd133c9f5c63d4d858fe3bcf7e2ec8f8fa47dbb9f0471 Sep 17 20:50:21.351787 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:21.351743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqdcz" event={"ID":"9bca1fc5-af86-4106-b5f5-b7ffdefab53a","Type":"ContainerStarted","Data":"1925d0a83bd14663ec8dd133c9f5c63d4d858fe3bcf7e2ec8f8fa47dbb9f0471"} Sep 17 20:50:21.459044 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:21.458905 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z556m"] Sep 17 20:50:21.461269 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:21.461246 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b744c7_94c6_400f_91eb_def724247bc0.slice/crio-4fc4dc9dc2901f29cac6c5d3509e07181cdfe1b1481ee0604c08ac9eec66c7aa WatchSource:0}: Error finding container 4fc4dc9dc2901f29cac6c5d3509e07181cdfe1b1481ee0604c08ac9eec66c7aa: Status 404 returned error can't find the container with id 4fc4dc9dc2901f29cac6c5d3509e07181cdfe1b1481ee0604c08ac9eec66c7aa Sep 17 20:50:22.261022 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:22.260988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:50:22.261481 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.261169 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 20:50:22.261481 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.261192 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b6c8df995-pbvm8: secret "image-registry-tls" not found Sep 17 20:50:22.261481 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.261269 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls podName:3d974800-a495-41e7-87e6-69e28de6156f nodeName:}" failed. No retries permitted until 2026-09-17 20:50:54.261248175 +0000 UTC m=+99.735930497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls") pod "image-registry-5b6c8df995-pbvm8" (UID: "3d974800-a495-41e7-87e6-69e28de6156f") : secret "image-registry-tls" not found Sep 17 20:50:22.356762 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:22.356715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z556m" event={"ID":"90b744c7-94c6-400f-91eb-def724247bc0","Type":"ContainerStarted","Data":"4fc4dc9dc2901f29cac6c5d3509e07181cdfe1b1481ee0604c08ac9eec66c7aa"} Sep 17 20:50:22.362140 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:22.362108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:50:22.362274 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.362252 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 20:50:22.362333 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.362307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls podName:74fbf86b-d686-4752-8c33-1607b699dece nodeName:}" failed. No retries permitted until 2026-09-17 20:50:54.362288794 +0000 UTC m=+99.836971125 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls") pod "dns-default-8jqfh" (UID: "74fbf86b-d686-4752-8c33-1607b699dece") : secret "dns-default-metrics-tls" not found Sep 17 20:50:22.866277 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:22.866236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:22.866488 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.866426 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:22.866546 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:22.866492 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls podName:2f0c9995-a015-4054-8ba6-0b39ae783f5e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:38.866473548 +0000 UTC m=+84.341155877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h9drm" (UID: "2f0c9995-a015-4054-8ba6-0b39ae783f5e") : secret "cluster-monitoring-operator-tls" not found Sep 17 20:50:23.067676 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:23.067641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:23.067841 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:23.067790 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 20:50:23.067841 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:23.067837 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls podName:89e0867e-07e3-4beb-97b3-6fabfe44cfbb nodeName:}" failed. No retries permitted until 2026-09-17 20:50:39.067824563 +0000 UTC m=+84.542506886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gs2qd" (UID: "89e0867e-07e3-4beb-97b3-6fabfe44cfbb") : secret "samples-operator-tls" not found Sep 17 20:50:26.369031 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.368978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z556m" event={"ID":"90b744c7-94c6-400f-91eb-def724247bc0","Type":"ContainerStarted","Data":"634943a07b527bcb2d4f89eb369a8191f95102fd22840cf389fc7f939c55350c"} Sep 17 20:50:26.369519 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.369085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:50:26.370947 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.370920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqdcz" event={"ID":"9bca1fc5-af86-4106-b5f5-b7ffdefab53a","Type":"ContainerStarted","Data":"126dd5027ad083a472631fb3b163f6264a7438567073e0d3b17e439a3b74c95d"} Sep 17 20:50:26.390986 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.390932 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z556m" podStartSLOduration=66.058138672 podStartE2EDuration="1m10.390916915s" totalImagesPullingTime="4.332778243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:26.39016113 +0000 UTC m=+71.864843494" watchObservedRunningTime="2026-09-17 20:50:26.390916915 +0000 UTC m=+71.865599287" Sep 17 20:50:26.409056 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.409011 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xqdcz" podStartSLOduration=65.781063172 podStartE2EDuration="1m10.408996666s" totalImagesPullingTime="4.627933494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:26.40830591 +0000 UTC m=+71.882988422" watchObservedRunningTime="2026-09-17 20:50:26.408996666 +0000 UTC m=+71.883679009" Sep 17 20:50:26.798258 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.798231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:26.798419 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.798310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:26.798419 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:26.798396 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 20:50:26.798536 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:26.798467 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls podName:1b6918ad-3c4a-445e-932a-61c856a69da7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:42.798446766 +0000 UTC m=+88.273129108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsw4z" (UID: "1b6918ad-3c4a-445e-932a-61c856a69da7") : secret "insights-runtime-extractor-tls" not found Sep 17 20:50:26.799344 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:26.799326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b6918ad-3c4a-445e-932a-61c856a69da7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:38.883049 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:38.882916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:38.885655 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:38.885629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f0c9995-a015-4054-8ba6-0b39ae783f5e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h9drm\" (UID: \"2f0c9995-a015-4054-8ba6-0b39ae783f5e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:38.948605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:38.947788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lrzjv\"" Sep 17 20:50:38.954972 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:38.954948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" Sep 17 20:50:39.084640 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.084610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:39.086950 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.086931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89e0867e-07e3-4beb-97b3-6fabfe44cfbb-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gs2qd\" (UID: \"89e0867e-07e3-4beb-97b3-6fabfe44cfbb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:39.103049 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.103027 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gnb5q\"" Sep 17 20:50:39.110649 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.110636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" Sep 17 20:50:39.246474 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.246446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm"] Sep 17 20:50:39.249066 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:39.249036 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0c9995_a015_4054_8ba6_0b39ae783f5e.slice/crio-39d9a3f42a415f0827407cf5dad06cbd49a2f42c5c7674c11bf77100208bbd14 WatchSource:0}: Error finding container 39d9a3f42a415f0827407cf5dad06cbd49a2f42c5c7674c11bf77100208bbd14: Status 404 returned error can't find the container with id 39d9a3f42a415f0827407cf5dad06cbd49a2f42c5c7674c11bf77100208bbd14 Sep 17 20:50:39.387561 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.387533 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd"] Sep 17 20:50:39.403460 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:39.403435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" event={"ID":"2f0c9995-a015-4054-8ba6-0b39ae783f5e","Type":"ContainerStarted","Data":"39d9a3f42a415f0827407cf5dad06cbd49a2f42c5c7674c11bf77100208bbd14"} Sep 17 20:50:40.407806 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:40.407773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" event={"ID":"89e0867e-07e3-4beb-97b3-6fabfe44cfbb","Type":"ContainerStarted","Data":"4c22814b88bbc3808f0d232d8025a602de24d4854a145912c34c028e8cd15ea8"} Sep 17 20:50:42.415807 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.415774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" event={"ID":"89e0867e-07e3-4beb-97b3-6fabfe44cfbb","Type":"ContainerStarted","Data":"c0ebbd36125e32fdfda31bcdc8d86fa3f5367934a844fe4ef3272c14d0e4118d"} Sep 17 20:50:42.416250 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.415834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" event={"ID":"89e0867e-07e3-4beb-97b3-6fabfe44cfbb","Type":"ContainerStarted","Data":"60870dd93d49ef20baae5c273e29ee6a974f49612327a4de1b79987912760345"} Sep 17 20:50:42.417302 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.417279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" event={"ID":"2f0c9995-a015-4054-8ba6-0b39ae783f5e","Type":"ContainerStarted","Data":"20263d64c00953df5f5867b58cce67e20ea8e1566322927be6112a25c930db18"} Sep 17 20:50:42.453562 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.453519 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gs2qd" podStartSLOduration=33.348540796 podStartE2EDuration="35.453505524s" totalImagesPullingTime="2.104964728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:42.453198544 +0000 UTC m=+87.927880887" watchObservedRunningTime="2026-09-17 20:50:42.453505524 +0000 UTC m=+87.928187867" Sep 17 20:50:42.485694 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.485655 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h9drm" podStartSLOduration=33.210173757 podStartE2EDuration="35.485642211s" totalImagesPullingTime="2.275468454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:42.48527874 +0000 UTC m=+87.959961083" watchObservedRunningTime="2026-09-17 20:50:42.485642211 +0000 UTC m=+87.960324554" Sep 17 20:50:42.811725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.811701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:42.813881 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.813859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b6918ad-3c4a-445e-932a-61c856a69da7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsw4z\" (UID: \"1b6918ad-3c4a-445e-932a-61c856a69da7\") " pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:42.844893 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.844872 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h4vct\"" Sep 17 20:50:42.850487 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:42.850474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsw4z" Sep 17 20:50:43.128654 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.128592 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rsw4z"] Sep 17 20:50:43.131517 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:43.131495 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6918ad_3c4a_445e_932a_61c856a69da7.slice/crio-d6b232bcdeb19f8cdad969e919f4a09d93ce8045e770a236a05278c5f439ef85 WatchSource:0}: Error finding container d6b232bcdeb19f8cdad969e919f4a09d93ce8045e770a236a05278c5f439ef85: Status 404 returned error can't find the container with id d6b232bcdeb19f8cdad969e919f4a09d93ce8045e770a236a05278c5f439ef85 Sep 17 20:50:43.422996 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.422909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsw4z" event={"ID":"1b6918ad-3c4a-445e-932a-61c856a69da7","Type":"ContainerStarted","Data":"df621d0fb506bdba2ee709dbb25b327859f17e8deef8ce8bc0ce360ad14149b0"} Sep 17 20:50:43.422996 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.422971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsw4z" event={"ID":"1b6918ad-3c4a-445e-932a-61c856a69da7","Type":"ContainerStarted","Data":"d6b232bcdeb19f8cdad969e919f4a09d93ce8045e770a236a05278c5f439ef85"} Sep 17 20:50:43.730356 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.730287 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm"] Sep 17 20:50:43.733963 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.733946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" Sep 17 20:50:43.752333 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.752312 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8"] Sep 17 20:50:43.755712 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.755685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.755831 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.755738 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pckbm\"" Sep 17 20:50:43.757327 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.757305 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm"] Sep 17 20:50:43.758156 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.758135 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc"] Sep 17 20:50:43.760215 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.760195 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 20:50:43.760322 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.760246 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 20:50:43.760400 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.760328 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xg76h\"" Sep 17 20:50:43.761559 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.761541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.765418 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.765398 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 17 20:50:43.765504 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.765433 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 20:50:43.765504 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.765440 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 20:50:43.765698 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.765685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lhw88\"" Sep 17 20:50:43.765740 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.765717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 20:50:43.818976 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.818956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk28r\" (UniqueName: \"kubernetes.io/projected/b978e1d5-6c80-44f7-ae67-c8c3329d7b7c-kube-api-access-xk28r\") pod \"network-check-source-7cbc45fddc-mzsbm\" (UID: \"b978e1d5-6c80-44f7-ae67-c8c3329d7b7c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" Sep 17 20:50:43.819066 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.819009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d643d2e6-72f0-470f-b84d-d225769d6bd5-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-6fhw8\" (UID: \"d643d2e6-72f0-470f-b84d-d225769d6bd5\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.819125 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.819069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d643d2e6-72f0-470f-b84d-d225769d6bd5-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-6fhw8\" (UID: \"d643d2e6-72f0-470f-b84d-d225769d6bd5\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.819125 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.819105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjxh\" (UniqueName: \"kubernetes.io/projected/15f24337-8050-4c3c-8e90-a881927014a1-kube-api-access-rwjxh\") pod \"managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc\" (UID: \"15f24337-8050-4c3c-8e90-a881927014a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.819202 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.819141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15f24337-8050-4c3c-8e90-a881927014a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc\" (UID: \"15f24337-8050-4c3c-8e90-a881927014a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.874440 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.874415 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc"] Sep 17 20:50:43.875549 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.875526 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp"] Sep 17 20:50:43.878925 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.878911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:43.883715 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.883693 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8"] Sep 17 20:50:43.904182 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.904166 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 20:50:43.907831 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.907815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bjhgj\"" Sep 17 20:50:43.919640 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.919614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk28r\" (UniqueName: \"kubernetes.io/projected/b978e1d5-6c80-44f7-ae67-c8c3329d7b7c-kube-api-access-xk28r\") pod \"network-check-source-7cbc45fddc-mzsbm\" (UID: \"b978e1d5-6c80-44f7-ae67-c8c3329d7b7c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" Sep 17 20:50:43.919730 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.919655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d643d2e6-72f0-470f-b84d-d225769d6bd5-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-6fhw8\" (UID: \"d643d2e6-72f0-470f-b84d-d225769d6bd5\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.919730 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.919688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d643d2e6-72f0-470f-b84d-d225769d6bd5-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-6fhw8\" (UID: \"d643d2e6-72f0-470f-b84d-d225769d6bd5\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.919895 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.919874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjxh\" (UniqueName: \"kubernetes.io/projected/15f24337-8050-4c3c-8e90-a881927014a1-kube-api-access-rwjxh\") pod \"managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc\" (UID: \"15f24337-8050-4c3c-8e90-a881927014a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.919949 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.919922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15f24337-8050-4c3c-8e90-a881927014a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc\" (UID: \"15f24337-8050-4c3c-8e90-a881927014a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.920309 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.920291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d643d2e6-72f0-470f-b84d-d225769d6bd5-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-6fhw8\" (UID: \"d643d2e6-72f0-470f-b84d-d225769d6bd5\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.922058 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.922039 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d643d2e6-72f0-470f-b84d-d225769d6bd5-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-6fhw8\" (UID: \"d643d2e6-72f0-470f-b84d-d225769d6bd5\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:43.922430 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.922411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15f24337-8050-4c3c-8e90-a881927014a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc\" (UID: \"15f24337-8050-4c3c-8e90-a881927014a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.954307 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.954281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk28r\" (UniqueName: \"kubernetes.io/projected/b978e1d5-6c80-44f7-ae67-c8c3329d7b7c-kube-api-access-xk28r\") pod \"network-check-source-7cbc45fddc-mzsbm\" (UID: \"b978e1d5-6c80-44f7-ae67-c8c3329d7b7c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" Sep 17 20:50:43.956075 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.956050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjxh\" (UniqueName: \"kubernetes.io/projected/15f24337-8050-4c3c-8e90-a881927014a1-kube-api-access-rwjxh\") pod \"managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc\" (UID: \"15f24337-8050-4c3c-8e90-a881927014a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:43.962986 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.962967 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b9bd7845-7jxfg"] Sep 17 20:50:43.967638 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.967618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:43.971855 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.971835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp"] Sep 17 20:50:43.973552 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.973530 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 20:50:43.973653 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.973563 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 20:50:43.974434 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.974331 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 20:50:43.974546 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.974512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 20:50:43.975179 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.975158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 20:50:43.976161 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.976143 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 20:50:43.976358 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.976344 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 20:50:43.976462 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:43.976440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5tlfx\"" Sep 17 20:50:44.020668 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.020648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/714314bb-6de6-47e0-ab08-d3ee0f4a2de1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-g4hhp\" (UID: \"714314bb-6de6-47e0-ab08-d3ee0f4a2de1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:44.042651 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.042629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" Sep 17 20:50:44.065013 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.064988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" Sep 17 20:50:44.067512 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.067494 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b9bd7845-7jxfg"] Sep 17 20:50:44.080400 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.080363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" Sep 17 20:50:44.121628 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9h6\" (UniqueName: \"kubernetes.io/projected/bedbe960-e01b-43b8-8e80-f20abee6c1a3-kube-api-access-4l9h6\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.121628 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-oauth-config\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.121882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/714314bb-6de6-47e0-ab08-d3ee0f4a2de1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-g4hhp\" (UID: \"714314bb-6de6-47e0-ab08-d3ee0f4a2de1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:44.121882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-oauth-serving-cert\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.121882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-service-ca\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.121882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-serving-cert\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.121882 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.121797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-config\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.124608 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.124585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/714314bb-6de6-47e0-ab08-d3ee0f4a2de1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-g4hhp\" (UID: \"714314bb-6de6-47e0-ab08-d3ee0f4a2de1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:44.186694 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.186675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:44.223725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.222939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-service-ca\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.223725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.222986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-serving-cert\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.223725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.223002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-config\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.223725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.223040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9h6\" (UniqueName: \"kubernetes.io/projected/bedbe960-e01b-43b8-8e80-f20abee6c1a3-kube-api-access-4l9h6\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.223725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.223058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-oauth-config\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.223725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.223087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-oauth-serving-cert\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.224116 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.223786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-oauth-serving-cert\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.224116 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.223790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-service-ca\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.224116 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.224105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-config\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.226183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.226163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-serving-cert\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.226263 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.226238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-oauth-config\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.259481 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.259458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9h6\" (UniqueName: \"kubernetes.io/projected/bedbe960-e01b-43b8-8e80-f20abee6c1a3-kube-api-access-4l9h6\") pod \"console-84b9bd7845-7jxfg\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.279589 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.279535 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:44.368071 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.368045 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm"] Sep 17 20:50:44.369033 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.369009 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8"] Sep 17 20:50:44.369177 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:44.369156 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd643d2e6_72f0_470f_b84d_d225769d6bd5.slice/crio-8de3e0045a929eb0c0089867ccfae4fcf1ef2404a253733852a91ced154753ca WatchSource:0}: Error finding container 8de3e0045a929eb0c0089867ccfae4fcf1ef2404a253733852a91ced154753ca: Status 404 returned error can't find the container with id 8de3e0045a929eb0c0089867ccfae4fcf1ef2404a253733852a91ced154753ca Sep 17 20:50:44.369868 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:44.369846 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb978e1d5_6c80_44f7_ae67_c8c3329d7b7c.slice/crio-0ee6748715dd3d2f0ce2ab11f35a0d54a391080601bf7c3f01c61d7d5dac734e WatchSource:0}: Error finding container 0ee6748715dd3d2f0ce2ab11f35a0d54a391080601bf7c3f01c61d7d5dac734e: Status 404 returned error can't find the container with id 0ee6748715dd3d2f0ce2ab11f35a0d54a391080601bf7c3f01c61d7d5dac734e Sep 17 20:50:44.379648 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.379625 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc"] Sep 17 20:50:44.382004 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:44.381979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15f24337_8050_4c3c_8e90_a881927014a1.slice/crio-867301ed12900aa0a241bad7907318483c646c047c851ccd37e0fcd54c1c5142 WatchSource:0}: Error finding container 867301ed12900aa0a241bad7907318483c646c047c851ccd37e0fcd54c1c5142: Status 404 returned error can't find the container with id 867301ed12900aa0a241bad7907318483c646c047c851ccd37e0fcd54c1c5142 Sep 17 20:50:44.426954 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.426927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" event={"ID":"15f24337-8050-4c3c-8e90-a881927014a1","Type":"ContainerStarted","Data":"867301ed12900aa0a241bad7907318483c646c047c851ccd37e0fcd54c1c5142"} Sep 17 20:50:44.428243 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.428222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" event={"ID":"d643d2e6-72f0-470f-b84d-d225769d6bd5","Type":"ContainerStarted","Data":"8de3e0045a929eb0c0089867ccfae4fcf1ef2404a253733852a91ced154753ca"} Sep 17 20:50:44.429789 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.429766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" event={"ID":"b978e1d5-6c80-44f7-ae67-c8c3329d7b7c","Type":"ContainerStarted","Data":"0ee6748715dd3d2f0ce2ab11f35a0d54a391080601bf7c3f01c61d7d5dac734e"} Sep 17 20:50:44.431496 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.431475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsw4z" event={"ID":"1b6918ad-3c4a-445e-932a-61c856a69da7","Type":"ContainerStarted","Data":"167878eab8426d3fec945a2d9eeefed453ef1ccb25ea3a0ae8a084ebb1998105"} Sep 17 20:50:44.469246 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.469226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp"] Sep 17 20:50:44.471747 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:44.471713 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714314bb_6de6_47e0_ab08_d3ee0f4a2de1.slice/crio-564ce8fc89c12960d69a62b260b4165e8c7d1d95a727614514bd28aa0bf92a2c WatchSource:0}: Error finding container 564ce8fc89c12960d69a62b260b4165e8c7d1d95a727614514bd28aa0bf92a2c: Status 404 returned error can't find the container with id 564ce8fc89c12960d69a62b260b4165e8c7d1d95a727614514bd28aa0bf92a2c Sep 17 20:50:44.508350 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.508324 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86db4d5c6f-v6bvn"] Sep 17 20:50:44.511263 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.511248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.523094 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.523075 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 20:50:44.531958 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.531907 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86db4d5c6f-v6bvn"] Sep 17 20:50:44.561121 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.561101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b9bd7845-7jxfg"] Sep 17 20:50:44.563939 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:44.563916 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedbe960_e01b_43b8_8e80_f20abee6c1a3.slice/crio-9721c17ffda8068af0548b6e3708255b5ff548277d23fe76cba3bfd60a270b82 WatchSource:0}: Error finding container 9721c17ffda8068af0548b6e3708255b5ff548277d23fe76cba3bfd60a270b82: Status 404 returned error can't find the container with id 9721c17ffda8068af0548b6e3708255b5ff548277d23fe76cba3bfd60a270b82 Sep 17 20:50:44.625999 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.625973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-oauth-serving-cert\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.626086 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.626054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-config\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.626152 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.626096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-oauth-config\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.626152 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.626121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhhd\" (UniqueName: \"kubernetes.io/projected/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-kube-api-access-qqhhd\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.626223 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.626187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-trusted-ca-bundle\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.626275 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.626257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-service-ca\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.626310 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.626284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-serving-cert\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.727629 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-config\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.727759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-oauth-config\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.727759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhhd\" (UniqueName: \"kubernetes.io/projected/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-kube-api-access-qqhhd\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.727759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-trusted-ca-bundle\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.727759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-service-ca\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.728010 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-serving-cert\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.728010 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.727791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-oauth-serving-cert\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.728348 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.728325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-config\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.728456 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.728395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-oauth-serving-cert\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.728699 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.728674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-service-ca\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.729086 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.729062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-trusted-ca-bundle\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.730733 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.730691 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-oauth-config\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.730900 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.730879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-serving-cert\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.744598 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.744574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhhd\" (UniqueName: \"kubernetes.io/projected/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-kube-api-access-qqhhd\") pod \"console-86db4d5c6f-v6bvn\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:44.820198 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:44.820146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:45.123958 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:45.123889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86db4d5c6f-v6bvn"] Sep 17 20:50:45.127263 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:45.127236 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d215d4c_d81f_440c_a3f7_d153cdecdd5b.slice/crio-6acccbf7c1f2a818ac5730190fbdedc85e164db1061cdfdd209bbcc9b215923c WatchSource:0}: Error finding container 6acccbf7c1f2a818ac5730190fbdedc85e164db1061cdfdd209bbcc9b215923c: Status 404 returned error can't find the container with id 6acccbf7c1f2a818ac5730190fbdedc85e164db1061cdfdd209bbcc9b215923c Sep 17 20:50:45.462594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:45.462427 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9bd7845-7jxfg" event={"ID":"bedbe960-e01b-43b8-8e80-f20abee6c1a3","Type":"ContainerStarted","Data":"9721c17ffda8068af0548b6e3708255b5ff548277d23fe76cba3bfd60a270b82"} Sep 17 20:50:45.472719 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:45.472565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" event={"ID":"714314bb-6de6-47e0-ab08-d3ee0f4a2de1","Type":"ContainerStarted","Data":"564ce8fc89c12960d69a62b260b4165e8c7d1d95a727614514bd28aa0bf92a2c"} Sep 17 20:50:45.480397 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:45.480262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" event={"ID":"b978e1d5-6c80-44f7-ae67-c8c3329d7b7c","Type":"ContainerStarted","Data":"f83eed06a8f231c8e970021adb320e7092af41891d8b62896e5b9f0d17bb6a3b"} Sep 17 20:50:45.486003 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:45.485954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86db4d5c6f-v6bvn" event={"ID":"5d215d4c-d81f-440c-a3f7-d153cdecdd5b","Type":"ContainerStarted","Data":"6acccbf7c1f2a818ac5730190fbdedc85e164db1061cdfdd209bbcc9b215923c"} Sep 17 20:50:45.505602 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:45.505556 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mzsbm" podStartSLOduration=2.505539277 podStartE2EDuration="2.505539277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:45.504758646 +0000 UTC m=+90.979440991" watchObservedRunningTime="2026-09-17 20:50:45.505539277 +0000 UTC m=+90.980221622" Sep 17 20:50:51.515518 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.515466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsw4z" event={"ID":"1b6918ad-3c4a-445e-932a-61c856a69da7","Type":"ContainerStarted","Data":"528368a85895a4aa18756896174fd0b4118e195208595088b3f499b4737e78ca"} Sep 17 20:50:51.517247 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.517222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86db4d5c6f-v6bvn" event={"ID":"5d215d4c-d81f-440c-a3f7-d153cdecdd5b","Type":"ContainerStarted","Data":"6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf"} Sep 17 20:50:51.518832 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.518810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9bd7845-7jxfg" event={"ID":"bedbe960-e01b-43b8-8e80-f20abee6c1a3","Type":"ContainerStarted","Data":"7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7"} Sep 17 20:50:51.520599 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.520579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" event={"ID":"714314bb-6de6-47e0-ab08-d3ee0f4a2de1","Type":"ContainerStarted","Data":"268a03a58ce6dfb9e47fa2726c1f4b19a48f66ec7952cfab8bb1c06f690a589f"} Sep 17 20:50:51.520746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.520731 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:51.522291 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.522263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" event={"ID":"15f24337-8050-4c3c-8e90-a881927014a1","Type":"ContainerStarted","Data":"3dc5c03276f73e39adb378065d7a66edf323e217c3d94dc835f87b5c4c0ab178"} Sep 17 20:50:51.523817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.523792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" event={"ID":"d643d2e6-72f0-470f-b84d-d225769d6bd5","Type":"ContainerStarted","Data":"b3bcbfc4474c0f9f1ba2e2814c141ae0ba673e75ea1c343fa30e809b07cbbaa6"} Sep 17 20:50:51.525836 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.525815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" Sep 17 20:50:51.536360 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.536326 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rsw4z" podStartSLOduration=32.653460205 podStartE2EDuration="40.536314817s" totalImagesPullingTime="7.882854612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:50:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:51.534917595 +0000 UTC m=+97.009599938" watchObservedRunningTime="2026-09-17 20:50:51.536314817 +0000 UTC m=+97.010997161" Sep 17 20:50:51.560923 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.560875 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-6fhw8" podStartSLOduration=1.7918128370000002 podStartE2EDuration="8.560861498s" totalImagesPullingTime="6.769048661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:51.560636277 +0000 UTC m=+97.035318640" watchObservedRunningTime="2026-09-17 20:50:51.560861498 +0000 UTC m=+97.035543842" Sep 17 20:50:51.576119 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.576084 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7999cd5bd6-qtvvc" podStartSLOduration=1.819927348 podStartE2EDuration="8.576071963s" totalImagesPullingTime="6.756144615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:51.575489087 +0000 UTC m=+97.050171431" watchObservedRunningTime="2026-09-17 20:50:51.576071963 +0000 UTC m=+97.050754306" Sep 17 20:50:51.598475 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.598428 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86db4d5c6f-v6bvn" podStartSLOduration=1.503013466 podStartE2EDuration="7.598411141s" totalImagesPullingTime="6.095397675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:51.597013169 +0000 UTC m=+97.071695515" watchObservedRunningTime="2026-09-17 20:50:51.598411141 +0000 UTC m=+97.073093486" Sep 17 20:50:51.614083 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.614049 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-g4hhp" podStartSLOduration=1.9485599260000002 podStartE2EDuration="8.614036831s" totalImagesPullingTime="6.665476905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:51.613263371 +0000 UTC m=+97.087945719" watchObservedRunningTime="2026-09-17 20:50:51.614036831 +0000 UTC m=+97.088719175" Sep 17 20:50:51.648947 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:51.648902 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b9bd7845-7jxfg" podStartSLOduration=2.00431499 podStartE2EDuration="8.648891534s" totalImagesPullingTime="6.644576544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:51.648801508 +0000 UTC m=+97.123483853" watchObservedRunningTime="2026-09-17 20:50:51.648891534 +0000 UTC m=+97.123573878" Sep 17 20:50:54.280551 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.280516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:54.280928 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.280583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:54.285437 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.285353 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:54.311886 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.311853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:50:54.314597 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.314569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"image-registry-5b6c8df995-pbvm8\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:50:54.330067 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.330048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p2pj8\"" Sep 17 20:50:54.335220 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.335206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:50:54.413027 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.413000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:50:54.415309 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.415289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74fbf86b-d686-4752-8c33-1607b699dece-metrics-tls\") pod \"dns-default-8jqfh\" (UID: \"74fbf86b-d686-4752-8c33-1607b699dece\") " pod="openshift-dns/dns-default-8jqfh" Sep 17 20:50:54.538424 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.538355 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:50:54.610604 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.610574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b6c8df995-pbvm8"] Sep 17 20:50:54.614624 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:54.614590 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d974800_a495_41e7_87e6_69e28de6156f.slice/crio-ebe3a8be6c8301e52f2f75e598f11405761dce9cd70e9897d5f9baf167135b52 WatchSource:0}: Error finding container ebe3a8be6c8301e52f2f75e598f11405761dce9cd70e9897d5f9baf167135b52: Status 404 returned error can't find the container with id ebe3a8be6c8301e52f2f75e598f11405761dce9cd70e9897d5f9baf167135b52 Sep 17 20:50:54.705626 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.705606 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6qzc\"" Sep 17 20:50:54.713272 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.713253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jqfh" Sep 17 20:50:54.820333 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.820268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:54.820451 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.820346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:54.825989 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.825968 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:54.985525 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:54.985496 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jqfh"] Sep 17 20:50:54.988874 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:54.988850 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fbf86b_d686_4752_8c33_1607b699dece.slice/crio-efa9b639a0e872e00b64126a0f1b12096cc5f29def573fa9ce5d307e2d8be591 WatchSource:0}: Error finding container efa9b639a0e872e00b64126a0f1b12096cc5f29def573fa9ce5d307e2d8be591: Status 404 returned error can't find the container with id efa9b639a0e872e00b64126a0f1b12096cc5f29def573fa9ce5d307e2d8be591 Sep 17 20:50:55.539541 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.539508 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" event={"ID":"3d974800-a495-41e7-87e6-69e28de6156f","Type":"ContainerStarted","Data":"930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967"} Sep 17 20:50:55.539916 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.539552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" event={"ID":"3d974800-a495-41e7-87e6-69e28de6156f","Type":"ContainerStarted","Data":"ebe3a8be6c8301e52f2f75e598f11405761dce9cd70e9897d5f9baf167135b52"} Sep 17 20:50:55.539916 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.539576 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:50:55.540833 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.540811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jqfh" event={"ID":"74fbf86b-d686-4752-8c33-1607b699dece","Type":"ContainerStarted","Data":"efa9b639a0e872e00b64126a0f1b12096cc5f29def573fa9ce5d307e2d8be591"} Sep 17 20:50:55.544529 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.544511 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:50:55.563839 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.563786 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" podStartSLOduration=99.563765529 podStartE2EDuration="1m39.563765529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:55.561655159 +0000 UTC m=+101.036337516" watchObservedRunningTime="2026-09-17 20:50:55.563765529 +0000 UTC m=+101.038447870" Sep 17 20:50:55.611722 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:55.611699 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b9bd7845-7jxfg"] Sep 17 20:50:57.376277 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.376247 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z556m" Sep 17 20:50:57.550592 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.550556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jqfh" event={"ID":"74fbf86b-d686-4752-8c33-1607b699dece","Type":"ContainerStarted","Data":"6e0e9be4c1fc3d209e3a77fa0bc5cf26913924f708253fa1ce03144e24635f15"} Sep 17 20:50:57.550725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.550616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jqfh" event={"ID":"74fbf86b-d686-4752-8c33-1607b699dece","Type":"ContainerStarted","Data":"56b6a1471e0ae1968f8ff3f6eb3b6ea793fc9305c6416572f2510944dad190d7"} Sep 17 20:50:57.550725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.550661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8jqfh" Sep 17 20:50:57.574162 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.574112 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8jqfh" podStartSLOduration=65.726385326 podStartE2EDuration="1m7.574100253s" totalImagesPullingTime="1.847714927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:57.571480941 +0000 UTC m=+103.046163287" watchObservedRunningTime="2026-09-17 20:50:57.574100253 +0000 UTC m=+103.048782626" Sep 17 20:50:57.582324 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.582297 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd"] Sep 17 20:50:57.585488 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.585474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.588582 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.588563 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 20:50:57.588824 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.588811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gtxd9\"" Sep 17 20:50:57.588941 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.588923 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 20:50:57.589020 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.588926 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 20:50:57.598462 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.598436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd"] Sep 17 20:50:57.631143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.631090 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8qccf"] Sep 17 20:50:57.634188 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.634173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.637533 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.637511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 20:50:57.637680 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.637662 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vfzf5\"" Sep 17 20:50:57.637789 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.637775 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 20:50:57.638187 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.638171 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 20:50:57.741630 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.741630 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.741782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-root\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.741782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-wtmp\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.741782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wl7n\" (UniqueName: \"kubernetes.io/projected/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-kube-api-access-9wl7n\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.741937 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.741937 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.741937 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.742034 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jz8z\" (UniqueName: \"kubernetes.io/projected/1c3157ff-c8e0-4144-9aee-7e12931611f7-kube-api-access-6jz8z\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.742034 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.741984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-sys\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.742034 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.742001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-textfile\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.742034 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.742016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-tls\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.742160 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.742103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c3157ff-c8e0-4144-9aee-7e12931611f7-metrics-client-ca\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.842872 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.842843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.842872 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.842873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-root\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843062 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.842894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-wtmp\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843062 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.842917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wl7n\" (UniqueName: \"kubernetes.io/projected/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-kube-api-access-9wl7n\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.843062 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.842947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843062 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.842978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-root\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843062 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz8z\" (UniqueName: \"kubernetes.io/projected/1c3157ff-c8e0-4144-9aee-7e12931611f7-kube-api-access-6jz8z\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-sys\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-wtmp\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c3157ff-c8e0-4144-9aee-7e12931611f7-sys\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-textfile\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-tls\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c3157ff-c8e0-4144-9aee-7e12931611f7-metrics-client-ca\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-textfile\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:57.843526 2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:57.843570 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:57.843591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-tls podName:fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e nodeName:}" failed. No retries permitted until 2026-09-17 20:50:58.343571678 +0000 UTC m=+103.818254017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-cx6dd" (UID: "fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e") : secret "openshift-state-metrics-tls" not found Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:57.843625 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-tls podName:1c3157ff-c8e0-4144-9aee-7e12931611f7 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:58.343606741 +0000 UTC m=+103.818289076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-tls") pod "node-exporter-8qccf" (UID: "1c3157ff-c8e0-4144-9aee-7e12931611f7") : secret "node-exporter-tls" not found Sep 17 20:50:57.843667 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.843967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.843967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.843916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c3157ff-c8e0-4144-9aee-7e12931611f7-metrics-client-ca\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.846018 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.845992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.846117 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.846053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:57.851720 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.851700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz8z\" (UniqueName: \"kubernetes.io/projected/1c3157ff-c8e0-4144-9aee-7e12931611f7-kube-api-access-6jz8z\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:57.852566 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:57.852548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wl7n\" (UniqueName: \"kubernetes.io/projected/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-kube-api-access-9wl7n\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:58.347769 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.347742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-tls\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:58.347931 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.347788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:58.350318 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.350291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cx6dd\" (UID: \"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:58.350444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.350323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c3157ff-c8e0-4144-9aee-7e12931611f7-node-exporter-tls\") pod \"node-exporter-8qccf\" (UID: \"1c3157ff-c8e0-4144-9aee-7e12931611f7\") " pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:58.493980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.493953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" Sep 17 20:50:58.543698 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.542523 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qccf" Sep 17 20:50:58.559470 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:58.559435 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3157ff_c8e0_4144_9aee_7e12931611f7.slice/crio-e6055e6e31c4cf2acf140d871e74ee1a04849c06dc315388d18386a609b72980 WatchSource:0}: Error finding container e6055e6e31c4cf2acf140d871e74ee1a04849c06dc315388d18386a609b72980: Status 404 returned error can't find the container with id e6055e6e31c4cf2acf140d871e74ee1a04849c06dc315388d18386a609b72980 Sep 17 20:50:58.688582 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.688514 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:50:58.728151 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.728122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:50:58.728268 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.728257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.731050 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731002 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 20:50:58.731273 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731077 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 20:50:58.731273 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731165 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 20:50:58.731468 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731279 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 20:50:58.731468 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731292 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 20:50:58.731468 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731309 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 20:50:58.731468 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731312 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 20:50:58.731753 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731539 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 20:50:58.731753 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.731601 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gdt2x\"" Sep 17 20:50:58.733127 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.733110 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 20:50:58.770743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.770722 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd"] Sep 17 20:50:58.773886 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:50:58.773865 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1a9dfc_e2f3_4746_98f9_dc6439bc3b4e.slice/crio-6bf3f507d91e8d384d78115ae73f7b4808a7be202c4e1850ca75755527873b9e WatchSource:0}: Error finding container 6bf3f507d91e8d384d78115ae73f7b4808a7be202c4e1850ca75755527873b9e: Status 404 returned error can't find the container with id 6bf3f507d91e8d384d78115ae73f7b4808a7be202c4e1850ca75755527873b9e Sep 17 20:50:58.852466 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-config-volume\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852552 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852552 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852632 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852632 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-web-config\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852721 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852721 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-config-out\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852839 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852872 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv222\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-kube-api-access-qv222\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852904 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.852933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.852906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953244 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953332 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv222\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-kube-api-access-qv222\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953332 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953332 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953538 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-config-volume\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953538 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953538 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953538 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953538 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-web-config\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953799 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953799 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953799 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.953799 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-config-out\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.954011 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:58.953815 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle podName:8160ae4c-99ef-4e49-af91-403cfa393667 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:59.453791823 +0000 UTC m=+104.928474150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667") : configmap references non-existent config key: ca-bundle.crt Sep 17 20:50:58.954011 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.953829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.954961 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:58.954871 2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 17 20:50:58.954961 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:50:58.954931 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls podName:8160ae4c-99ef-4e49-af91-403cfa393667 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:59.454913619 +0000 UTC m=+104.929595946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667") : secret "alertmanager-main-tls" not found Sep 17 20:50:58.955230 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.955206 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.956519 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.956466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.956891 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.956865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-config-volume\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.956991 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.956972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.957156 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.957134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-config-out\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.957303 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.957283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.957345 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.957333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.957920 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.957899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.958049 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.958033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-web-config\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:58.962618 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:58.962595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv222\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-kube-api-access-qv222\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:59.458038 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.458006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:59.458197 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.458052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:59.458815 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.458791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:59.460537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.460514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:59.564566 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.564530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" event={"ID":"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e","Type":"ContainerStarted","Data":"4c59a0edda2d9b0f6b6a688413ad07293eb21a3aa937a4e031276f5993551c32"} Sep 17 20:50:59.564933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.564591 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" event={"ID":"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e","Type":"ContainerStarted","Data":"5ed2d0fddbab8a0a0681f0f080f26ff504aa6d921af5366ced0773cbce62fc54"} Sep 17 20:50:59.564933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.564615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" event={"ID":"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e","Type":"ContainerStarted","Data":"6bf3f507d91e8d384d78115ae73f7b4808a7be202c4e1850ca75755527873b9e"} Sep 17 20:50:59.565776 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.565753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qccf" event={"ID":"1c3157ff-c8e0-4144-9aee-7e12931611f7","Type":"ContainerStarted","Data":"e6055e6e31c4cf2acf140d871e74ee1a04849c06dc315388d18386a609b72980"} Sep 17 20:50:59.643772 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.643741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:50:59.953252 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:50:59.953226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:51:00.141457 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:51:00.141426 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8160ae4c_99ef_4e49_af91_403cfa393667.slice/crio-b8b48cecf81257e6818769888b9995487acf3d4842d4e1facef27b89e279d269 WatchSource:0}: Error finding container b8b48cecf81257e6818769888b9995487acf3d4842d4e1facef27b89e279d269: Status 404 returned error can't find the container with id b8b48cecf81257e6818769888b9995487acf3d4842d4e1facef27b89e279d269 Sep 17 20:51:00.570604 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:00.570567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"b8b48cecf81257e6818769888b9995487acf3d4842d4e1facef27b89e279d269"} Sep 17 20:51:01.576543 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:01.576505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" event={"ID":"fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e","Type":"ContainerStarted","Data":"8ffc39de0d5f33a546fd4132034aea62b985f01505533672bae2ac6404a9de9c"} Sep 17 20:51:01.578269 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:01.578243 2579 generic.go:356] "Generic (PLEG): container finished" podID="1c3157ff-c8e0-4144-9aee-7e12931611f7" containerID="2a6f9582f327d322f0e0b1a8217de616d6076364dd9c69ed569b118e133f5b97" exitCode=0 Sep 17 20:51:01.578488 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:01.578295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qccf" event={"ID":"1c3157ff-c8e0-4144-9aee-7e12931611f7","Type":"ContainerDied","Data":"2a6f9582f327d322f0e0b1a8217de616d6076364dd9c69ed569b118e133f5b97"} Sep 17 20:51:01.597901 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:01.597001 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cx6dd" podStartSLOduration=2.8699935720000003 podStartE2EDuration="4.596975932s" totalImagesPullingTime="1.72698236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:01.595587072 +0000 UTC m=+107.070269416" watchObservedRunningTime="2026-09-17 20:51:01.596975932 +0000 UTC m=+107.071658277" Sep 17 20:51:02.405832 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.405802 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b"] Sep 17 20:51:02.408588 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.408573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:02.411135 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.411111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 17 20:51:02.411234 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.411150 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mjlps\"" Sep 17 20:51:02.417716 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.417694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b"] Sep 17 20:51:02.481087 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.481063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad16d675-8e2d-4f9c-be85-4933217f2f81-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:02.481193 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.481099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsnl\" (UniqueName: \"kubernetes.io/projected/ad16d675-8e2d-4f9c-be85-4933217f2f81-kube-api-access-5hsnl\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:02.583614 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.583586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad16d675-8e2d-4f9c-be85-4933217f2f81-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:02.583989 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.583652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsnl\" (UniqueName: \"kubernetes.io/projected/ad16d675-8e2d-4f9c-be85-4933217f2f81-kube-api-access-5hsnl\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:02.583989 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:51:02.583691 2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 17 20:51:02.583989 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:51:02.583786 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad16d675-8e2d-4f9c-be85-4933217f2f81-monitoring-plugin-cert podName:ad16d675-8e2d-4f9c-be85-4933217f2f81 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:03.083768271 +0000 UTC m=+108.558450598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ad16d675-8e2d-4f9c-be85-4933217f2f81-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-rxx9b" (UID: "ad16d675-8e2d-4f9c-be85-4933217f2f81") : secret "monitoring-plugin-cert" not found Sep 17 20:51:02.586640 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.586613 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22" exitCode=0 Sep 17 20:51:02.586727 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.586692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22"} Sep 17 20:51:02.589089 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.589066 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qccf" event={"ID":"1c3157ff-c8e0-4144-9aee-7e12931611f7","Type":"ContainerStarted","Data":"57dea0d8911e8dc984a43faa602490865af3d2a3ec857c1aadc2392f1097a5fd"} Sep 17 20:51:02.589171 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.589106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qccf" event={"ID":"1c3157ff-c8e0-4144-9aee-7e12931611f7","Type":"ContainerStarted","Data":"35c8836ee4bba93a0ef940fe5f1524bf84624911bea2b3e8b62c57330166b7b1"} Sep 17 20:51:02.596885 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.596868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsnl\" (UniqueName: \"kubernetes.io/projected/ad16d675-8e2d-4f9c-be85-4933217f2f81-kube-api-access-5hsnl\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:02.657870 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:02.657788 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8qccf" podStartSLOduration=3.343785522 podStartE2EDuration="5.657777741s" totalImagesPullingTime="2.225665097s" totalInitContainerRuntime="88.327122ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:02.657483966 +0000 UTC m=+108.132166309" watchObservedRunningTime="2026-09-17 20:51:02.657777741 +0000 UTC m=+108.132460086" Sep 17 20:51:03.087418 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.087381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad16d675-8e2d-4f9c-be85-4933217f2f81-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:03.089881 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.089857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad16d675-8e2d-4f9c-be85-4933217f2f81-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-rxx9b\" (UID: \"ad16d675-8e2d-4f9c-be85-4933217f2f81\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:03.317092 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.317050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:03.593671 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.593600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b"] Sep 17 20:51:03.596192 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:51:03.596148 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad16d675_8e2d_4f9c_be85_4933217f2f81.slice/crio-e6afe5fe9d1b47b5e0d23199be7ef98dff2605e78712b9f2893b1b1ec7407bbf WatchSource:0}: Error finding container e6afe5fe9d1b47b5e0d23199be7ef98dff2605e78712b9f2893b1b1ec7407bbf: Status 404 returned error can't find the container with id e6afe5fe9d1b47b5e0d23199be7ef98dff2605e78712b9f2893b1b1ec7407bbf Sep 17 20:51:03.794754 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.794727 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:51:03.798599 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.798580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.802354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.802324 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 20:51:03.802531 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.802412 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 20:51:03.803315 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.803288 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 20:51:03.803315 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.803294 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 20:51:03.803841 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.803819 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 20:51:03.806707 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.805699 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 20:51:03.806707 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.805844 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 20:51:03.806707 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.806594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 20:51:03.807303 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.807239 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 20:51:03.807452 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.807431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-57f9qld6qn70h\"" Sep 17 20:51:03.809048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.808128 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 20:51:03.809048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.808241 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 20:51:03.809048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.808639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 20:51:03.809048 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.808903 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 20:51:03.809557 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.809536 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5pvrq\"" Sep 17 20:51:03.813467 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.813422 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 20:51:03.815232 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.815210 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 20:51:03.826545 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.826520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:51:03.893599 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893599 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-config-out\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-config\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.893783 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcp2v\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-kube-api-access-gcp2v\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-web-config\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.893987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.894004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.894183 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.894028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995028 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.994997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995186 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-config-out\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995186 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995304 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-config\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995304 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995422 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995422 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995422 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995574 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.995987 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.995955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996096 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996096 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcp2v\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-kube-api-access-gcp2v\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996096 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996275 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996275 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-web-config\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996275 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996461 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996461 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996461 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996461 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996653 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996653 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.996759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.996725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.997458 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.997428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.997967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.997942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.998210 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.998182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.999280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.999253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.999452 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.999427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:03.999864 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:03.999828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-config\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.000326 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.000291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-config-out\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.001336 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.000792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.001336 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.001186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.002021 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.001711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.002021 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.001974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-web-config\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.002192 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.002171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.002479 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.002459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.002559 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.002516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.002817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.002800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.003983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.003962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.010725 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.010707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcp2v\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-kube-api-access-gcp2v\") pod \"prometheus-k8s-0\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.110241 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.110177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:04.435447 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.435355 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:51:04.600224 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.600192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" event={"ID":"ad16d675-8e2d-4f9c-be85-4933217f2f81","Type":"ContainerStarted","Data":"e6afe5fe9d1b47b5e0d23199be7ef98dff2605e78712b9f2893b1b1ec7407bbf"} Sep 17 20:51:04.603450 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.603430 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00"} Sep 17 20:51:04.603575 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.603486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110"} Sep 17 20:51:04.603575 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.603514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f"} Sep 17 20:51:04.603575 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.603547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f"} Sep 17 20:51:04.603575 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.603571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58"} Sep 17 20:51:04.604847 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.604827 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" exitCode=0 Sep 17 20:51:04.604931 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.604858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} Sep 17 20:51:04.604931 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:04.604903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"f0564e6d0d8188301c623849581d92ae8550d2e0a99f444b90287fa9405f2a3e"} Sep 17 20:51:06.614983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.614893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" event={"ID":"ad16d675-8e2d-4f9c-be85-4933217f2f81","Type":"ContainerStarted","Data":"4fbc74d634a91e798d07513f11876ddbaa5d2873ec7316e0b0a645510882f6e6"} Sep 17 20:51:06.614983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.614960 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:06.618888 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.618850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerStarted","Data":"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3"} Sep 17 20:51:06.620264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.620240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" Sep 17 20:51:06.634067 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.634018 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-rxx9b" podStartSLOduration=2.605750873 podStartE2EDuration="4.634000576s" totalImagesPullingTime="2.028249703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:06.631021175 +0000 UTC m=+112.105703519" watchObservedRunningTime="2026-09-17 20:51:06.634000576 +0000 UTC m=+112.108682921" Sep 17 20:51:06.678175 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.678127 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.112001206 podStartE2EDuration="8.678112062s" totalImagesPullingTime="4.549489425s" totalInitContainerRuntime="16.621431ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:06.675597109 +0000 UTC m=+112.150279454" watchObservedRunningTime="2026-09-17 20:51:06.678112062 +0000 UTC m=+112.152794405" Sep 17 20:51:06.725986 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:06.725955 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b6c8df995-pbvm8"] Sep 17 20:51:07.562207 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:07.562172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8jqfh" Sep 17 20:51:08.631598 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:08.631560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} Sep 17 20:51:08.631980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:08.631673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} Sep 17 20:51:10.643643 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:10.643608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} Sep 17 20:51:10.644075 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:10.643723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} Sep 17 20:51:10.644075 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:10.643777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} Sep 17 20:51:10.644075 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:10.643817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerStarted","Data":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} Sep 17 20:51:10.679106 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:10.679051 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.543027608 podStartE2EDuration="7.679030207s" totalImagesPullingTime="5.117917035s" totalInitContainerRuntime="18.085564ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:10.678423643 +0000 UTC m=+116.153106007" watchObservedRunningTime="2026-09-17 20:51:10.679030207 +0000 UTC m=+116.153712553" Sep 17 20:51:14.110793 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:14.110745 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:51:16.731946 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:16.731919 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:51:21.565843 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:21.565774 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84b9bd7845-7jxfg" podUID="bedbe960-e01b-43b8-8e80-f20abee6c1a3" containerName="console" containerID="cri-o://7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7" gracePeriod=15 Sep 17 20:51:22.346738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.346718 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:51:22.447656 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.447633 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-service-ca\") pod \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " Sep 17 20:51:22.447782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.447668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-config\") pod \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " Sep 17 20:51:22.447782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.447699 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-oauth-config\") pod \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " Sep 17 20:51:22.447782 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.447723 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-oauth-serving-cert\") pod \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " Sep 17 20:51:22.447886 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.447851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-serving-cert\") pod \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " Sep 17 20:51:22.447984 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.447961 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l9h6\" (UniqueName: \"kubernetes.io/projected/bedbe960-e01b-43b8-8e80-f20abee6c1a3-kube-api-access-4l9h6\") pod \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\" (UID: \"bedbe960-e01b-43b8-8e80-f20abee6c1a3\") " Sep 17 20:51:22.448108 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.448068 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-service-ca" (OuterVolumeSpecName: "service-ca") pod "bedbe960-e01b-43b8-8e80-f20abee6c1a3" (UID: "bedbe960-e01b-43b8-8e80-f20abee6c1a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:22.448108 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.448064 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bedbe960-e01b-43b8-8e80-f20abee6c1a3" (UID: "bedbe960-e01b-43b8-8e80-f20abee6c1a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:22.448108 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.448077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-config" (OuterVolumeSpecName: "console-config") pod "bedbe960-e01b-43b8-8e80-f20abee6c1a3" (UID: "bedbe960-e01b-43b8-8e80-f20abee6c1a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:22.448262 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.448229 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-service-ca\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:22.448262 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.448242 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:22.448262 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.448250 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bedbe960-e01b-43b8-8e80-f20abee6c1a3-oauth-serving-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:22.450147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.450095 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bedbe960-e01b-43b8-8e80-f20abee6c1a3" (UID: "bedbe960-e01b-43b8-8e80-f20abee6c1a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:51:22.450147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.450115 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bedbe960-e01b-43b8-8e80-f20abee6c1a3" (UID: "bedbe960-e01b-43b8-8e80-f20abee6c1a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:51:22.450242 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.450207 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedbe960-e01b-43b8-8e80-f20abee6c1a3-kube-api-access-4l9h6" (OuterVolumeSpecName: "kube-api-access-4l9h6") pod "bedbe960-e01b-43b8-8e80-f20abee6c1a3" (UID: "bedbe960-e01b-43b8-8e80-f20abee6c1a3"). InnerVolumeSpecName "kube-api-access-4l9h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:51:22.548713 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.548689 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9h6\" (UniqueName: \"kubernetes.io/projected/bedbe960-e01b-43b8-8e80-f20abee6c1a3-kube-api-access-4l9h6\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:22.548713 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.548710 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-oauth-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:22.548834 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.548720 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bedbe960-e01b-43b8-8e80-f20abee6c1a3-console-serving-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:22.693844 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.693814 2579 generic.go:356] "Generic (PLEG): container finished" podID="bedbe960-e01b-43b8-8e80-f20abee6c1a3" containerID="7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7" exitCode=0 Sep 17 20:51:22.694233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.693865 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b9bd7845-7jxfg" Sep 17 20:51:22.694233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.693894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9bd7845-7jxfg" event={"ID":"bedbe960-e01b-43b8-8e80-f20abee6c1a3","Type":"ContainerDied","Data":"7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7"} Sep 17 20:51:22.694233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.693958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b9bd7845-7jxfg" event={"ID":"bedbe960-e01b-43b8-8e80-f20abee6c1a3","Type":"ContainerDied","Data":"9721c17ffda8068af0548b6e3708255b5ff548277d23fe76cba3bfd60a270b82"} Sep 17 20:51:22.694233 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.693979 2579 scope.go:117] "RemoveContainer" containerID="7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7" Sep 17 20:51:22.705033 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.705016 2579 scope.go:117] "RemoveContainer" containerID="7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7" Sep 17 20:51:22.705273 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:51:22.705253 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7\": container with ID starting with 7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7 not found: ID does not exist" containerID="7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7" Sep 17 20:51:22.705323 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.705281 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7"} err="failed to get container status \"7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7\": rpc error: code = NotFound desc = could not find container \"7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7\": container with ID starting with 7fd545a81d97f03a6ebcc46de7d76afbc568050e64ad3bfeba59fc82958a05f7 not found: ID does not exist" Sep 17 20:51:22.719836 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.719817 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b9bd7845-7jxfg"] Sep 17 20:51:22.721349 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:22.721330 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b9bd7845-7jxfg"] Sep 17 20:51:23.052189 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:23.052164 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bedbe960-e01b-43b8-8e80-f20abee6c1a3" path="/var/lib/kubelet/pods/bedbe960-e01b-43b8-8e80-f20abee6c1a3/volumes" Sep 17 20:51:24.765654 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:24.765613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:51:24.768421 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:24.768395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85d8f30-029c-4596-a39d-101227da18aa-metrics-certs\") pod \"network-metrics-daemon-f8sff\" (UID: \"e85d8f30-029c-4596-a39d-101227da18aa\") " pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:51:24.930522 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:24.930487 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86db4d5c6f-v6bvn"] Sep 17 20:51:25.070157 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:25.070089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-775ms\"" Sep 17 20:51:25.077868 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:25.077850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8sff" Sep 17 20:51:25.360627 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:25.360566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f8sff"] Sep 17 20:51:25.362759 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:51:25.362733 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85d8f30_029c_4596_a39d_101227da18aa.slice/crio-f449997945027053f449e04d78197cde3266fa898f00b812767f021888db4c74 WatchSource:0}: Error finding container f449997945027053f449e04d78197cde3266fa898f00b812767f021888db4c74: Status 404 returned error can't find the container with id f449997945027053f449e04d78197cde3266fa898f00b812767f021888db4c74 Sep 17 20:51:25.709248 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:25.709183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f8sff" event={"ID":"e85d8f30-029c-4596-a39d-101227da18aa","Type":"ContainerStarted","Data":"f449997945027053f449e04d78197cde3266fa898f00b812767f021888db4c74"} Sep 17 20:51:26.714890 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:26.714854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f8sff" event={"ID":"e85d8f30-029c-4596-a39d-101227da18aa","Type":"ContainerStarted","Data":"75e03275434af6ca28a8a42134021a2ff76800b50f392dd896db39105011c9ba"} Sep 17 20:51:26.715286 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:26.714907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f8sff" event={"ID":"e85d8f30-029c-4596-a39d-101227da18aa","Type":"ContainerStarted","Data":"0086b4632611a7567c792061e8654779ac51be9c85441cc0d86b3e563609a005"} Sep 17 20:51:26.752651 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:26.752572 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f8sff" podStartSLOduration=129.649352849 podStartE2EDuration="2m10.752556548s" totalImagesPullingTime="1.103203699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:26.750773367 +0000 UTC m=+132.225455713" watchObservedRunningTime="2026-09-17 20:51:26.752556548 +0000 UTC m=+132.227238929" Sep 17 20:51:30.732489 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:30.732407 2579 generic.go:356] "Generic (PLEG): container finished" podID="8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9" containerID="8f6beddac2297c969d8000d460e303bcef2c659a891cdce02b80288804699998" exitCode=0 Sep 17 20:51:30.732489 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:30.732475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" event={"ID":"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9","Type":"ContainerDied","Data":"8f6beddac2297c969d8000d460e303bcef2c659a891cdce02b80288804699998"} Sep 17 20:51:30.732886 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:30.732760 2579 scope.go:117] "RemoveContainer" containerID="8f6beddac2297c969d8000d460e303bcef2c659a891cdce02b80288804699998" Sep 17 20:51:31.738360 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:31.738328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-shcng" event={"ID":"8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9","Type":"ContainerStarted","Data":"6a6f5a62d94ccc60b8968c8554766d17645a602176ca62803756805543b9f65b"} Sep 17 20:51:31.748106 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:31.748077 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" podUID="3d974800-a495-41e7-87e6-69e28de6156f" containerName="registry" containerID="cri-o://930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967" gracePeriod=30 Sep 17 20:51:31.991779 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:31.991730 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:51:32.124814 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.124769 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-image-registry-private-configuration\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.124980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.124823 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-registry-certificates\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.124980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.124859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-installation-pull-secrets\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.124980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.124899 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-bound-sa-token\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.124980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.124947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-trusted-ca\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.124980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.124975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d974800-a495-41e7-87e6-69e28de6156f-ca-trust-extracted\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.125224 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.125004 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z5tf\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-kube-api-access-6z5tf\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.125224 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.125055 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") pod \"3d974800-a495-41e7-87e6-69e28de6156f\" (UID: \"3d974800-a495-41e7-87e6-69e28de6156f\") " Sep 17 20:51:32.125336 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.125304 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:32.125459 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.125433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:32.127594 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.127555 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:51:32.127819 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.127789 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:51:32.127938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.127887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-kube-api-access-6z5tf" (OuterVolumeSpecName: "kube-api-access-6z5tf") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "kube-api-access-6z5tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:51:32.128003 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.127955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:51:32.128003 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.127983 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:51:32.134337 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.134315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d974800-a495-41e7-87e6-69e28de6156f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3d974800-a495-41e7-87e6-69e28de6156f" (UID: "3d974800-a495-41e7-87e6-69e28de6156f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 20:51:32.226024 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226000 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-bound-sa-token\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226024 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226022 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-trusted-ca\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226030 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d974800-a495-41e7-87e6-69e28de6156f-ca-trust-extracted\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226041 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5tf\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-kube-api-access-6z5tf\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226050 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d974800-a495-41e7-87e6-69e28de6156f-registry-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226059 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-image-registry-private-configuration\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226067 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d974800-a495-41e7-87e6-69e28de6156f-registry-certificates\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.226143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.226076 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d974800-a495-41e7-87e6-69e28de6156f-installation-pull-secrets\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:32.743736 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.743695 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d974800-a495-41e7-87e6-69e28de6156f" containerID="930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967" exitCode=0 Sep 17 20:51:32.744143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.743757 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" Sep 17 20:51:32.744143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.743782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" event={"ID":"3d974800-a495-41e7-87e6-69e28de6156f","Type":"ContainerDied","Data":"930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967"} Sep 17 20:51:32.744143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.743860 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b6c8df995-pbvm8" event={"ID":"3d974800-a495-41e7-87e6-69e28de6156f","Type":"ContainerDied","Data":"ebe3a8be6c8301e52f2f75e598f11405761dce9cd70e9897d5f9baf167135b52"} Sep 17 20:51:32.744143 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.743882 2579 scope.go:117] "RemoveContainer" containerID="930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967" Sep 17 20:51:32.755055 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.754858 2579 scope.go:117] "RemoveContainer" containerID="930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967" Sep 17 20:51:32.755407 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:51:32.755359 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967\": container with ID starting with 930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967 not found: ID does not exist" containerID="930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967" Sep 17 20:51:32.755480 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.755418 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967"} err="failed to get container status \"930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967\": rpc error: code = NotFound desc = could not find container \"930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967\": container with ID starting with 930feedc862cffdaadda721cb0c3ad9a5e749bae8c8af7e555f15ac0017b0967 not found: ID does not exist" Sep 17 20:51:32.775153 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.775132 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b6c8df995-pbvm8"] Sep 17 20:51:32.778432 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:32.778413 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b6c8df995-pbvm8"] Sep 17 20:51:33.053872 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:33.053843 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d974800-a495-41e7-87e6-69e28de6156f" path="/var/lib/kubelet/pods/3d974800-a495-41e7-87e6-69e28de6156f/volumes" Sep 17 20:51:49.950537 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:49.950481 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86db4d5c6f-v6bvn" podUID="5d215d4c-d81f-440c-a3f7-d153cdecdd5b" containerName="console" containerID="cri-o://6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf" gracePeriod=15 Sep 17 20:51:50.197743 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.197721 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:51:50.267139 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267112 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqhhd\" (UniqueName: \"kubernetes.io/projected/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-kube-api-access-qqhhd\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267139 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267140 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-serving-cert\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267346 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-oauth-config\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267346 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-config\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267346 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-oauth-serving-cert\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267346 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-service-ca\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267346 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267269 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-trusted-ca-bundle\") pod \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\" (UID: \"5d215d4c-d81f-440c-a3f7-d153cdecdd5b\") " Sep 17 20:51:50.267764 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:50.267764 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267722 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:50.267883 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-config" (OuterVolumeSpecName: "console-config") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:50.267883 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.267793 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:51:50.269496 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.269469 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:51:50.269814 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.269791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:51:50.269897 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.269810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-kube-api-access-qqhhd" (OuterVolumeSpecName: "kube-api-access-qqhhd") pod "5d215d4c-d81f-440c-a3f7-d153cdecdd5b" (UID: "5d215d4c-d81f-440c-a3f7-d153cdecdd5b"). InnerVolumeSpecName "kube-api-access-qqhhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:51:50.368140 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368120 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqhhd\" (UniqueName: \"kubernetes.io/projected/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-kube-api-access-qqhhd\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.368140 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368140 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-serving-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.368264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368149 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-oauth-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.368264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368159 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-console-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.368264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368167 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-oauth-serving-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.368264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368175 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-service-ca\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.368264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.368184 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d215d4c-d81f-440c-a3f7-d153cdecdd5b-trusted-ca-bundle\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:51:50.822490 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.822458 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d215d4c-d81f-440c-a3f7-d153cdecdd5b" containerID="6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf" exitCode=0 Sep 17 20:51:50.822612 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.822507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86db4d5c6f-v6bvn" event={"ID":"5d215d4c-d81f-440c-a3f7-d153cdecdd5b","Type":"ContainerDied","Data":"6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf"} Sep 17 20:51:50.822612 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.822511 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86db4d5c6f-v6bvn" Sep 17 20:51:50.822612 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.822551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86db4d5c6f-v6bvn" event={"ID":"5d215d4c-d81f-440c-a3f7-d153cdecdd5b","Type":"ContainerDied","Data":"6acccbf7c1f2a818ac5730190fbdedc85e164db1061cdfdd209bbcc9b215923c"} Sep 17 20:51:50.822612 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.822574 2579 scope.go:117] "RemoveContainer" containerID="6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf" Sep 17 20:51:50.831891 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.831873 2579 scope.go:117] "RemoveContainer" containerID="6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf" Sep 17 20:51:50.832217 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:51:50.832197 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf\": container with ID starting with 6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf not found: ID does not exist" containerID="6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf" Sep 17 20:51:50.832268 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.832226 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf"} err="failed to get container status \"6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf\": rpc error: code = NotFound desc = could not find container \"6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf\": container with ID starting with 6112824071ed8145d0908235d1d924f313ee05e3cb88d98d7e45cf8069625fbf not found: ID does not exist" Sep 17 20:51:50.844556 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.844532 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86db4d5c6f-v6bvn"] Sep 17 20:51:50.848491 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:50.848468 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86db4d5c6f-v6bvn"] Sep 17 20:51:51.052566 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:51:51.052529 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d215d4c-d81f-440c-a3f7-d153cdecdd5b" path="/var/lib/kubelet/pods/5d215d4c-d81f-440c-a3f7-d153cdecdd5b/volumes" Sep 17 20:52:04.111464 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:04.111428 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:04.132278 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:04.132246 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:04.901657 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:04.901626 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:18.056285 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056245 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:52:18.056720 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056587 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="alertmanager" containerID="cri-o://68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58" gracePeriod=120 Sep 17 20:52:18.056720 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056628 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-metric" containerID="cri-o://8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00" gracePeriod=120 Sep 17 20:52:18.056838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056717 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-web" containerID="cri-o://51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f" gracePeriod=120 Sep 17 20:52:18.056838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056745 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy" containerID="cri-o://3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110" gracePeriod=120 Sep 17 20:52:18.056838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056738 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="config-reloader" containerID="cri-o://f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f" gracePeriod=120 Sep 17 20:52:18.056988 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.056698 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="prom-label-proxy" containerID="cri-o://ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3" gracePeriod=120 Sep 17 20:52:18.945221 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945194 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3" exitCode=0 Sep 17 20:52:18.945221 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945217 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110" exitCode=0 Sep 17 20:52:18.945221 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945223 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f" exitCode=0 Sep 17 20:52:18.945221 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945228 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58" exitCode=0 Sep 17 20:52:18.945490 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945265 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3"} Sep 17 20:52:18.945490 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110"} Sep 17 20:52:18.945490 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f"} Sep 17 20:52:18.945490 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:18.945396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58"} Sep 17 20:52:19.301393 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.301355 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:19.478157 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478157 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478133 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-web\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478159 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv222\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-kube-api-access-qv222\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478188 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-metrics-client-ca\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478210 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-main-db\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478228 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478257 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-cluster-tls-config\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478286 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-config-out\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-web-config\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478765 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478389 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-config-volume\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478765 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478441 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.478765 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478735 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:19.478883 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.478806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:19.479289 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.479239 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 20:52:19.479530 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.479506 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-tls-assets\") pod \"8160ae4c-99ef-4e49-af91-403cfa393667\" (UID: \"8160ae4c-99ef-4e49-af91-403cfa393667\") " Sep 17 20:52:19.479840 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.479773 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.479840 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.479799 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8160ae4c-99ef-4e49-af91-403cfa393667-metrics-client-ca\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.479840 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.479813 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-alertmanager-main-db\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.481153 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.481112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.481274 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.481176 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-kube-api-access-qv222" (OuterVolumeSpecName: "kube-api-access-qv222") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "kube-api-access-qv222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:52:19.481681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.481660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.482467 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.482440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-config-volume" (OuterVolumeSpecName: "config-volume") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.482558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.482464 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-config-out" (OuterVolumeSpecName: "config-out") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 20:52:19.482558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.482540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.482983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.482956 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.483268 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.483253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:52:19.486863 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.486838 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.493689 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.493669 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-web-config" (OuterVolumeSpecName: "web-config") pod "8160ae4c-99ef-4e49-af91-403cfa393667" (UID: "8160ae4c-99ef-4e49-af91-403cfa393667"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:19.580823 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580800 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-tls-assets\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580823 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580821 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580832 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580843 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv222\" (UniqueName: \"kubernetes.io/projected/8160ae4c-99ef-4e49-af91-403cfa393667-kube-api-access-qv222\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580852 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-main-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580860 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-cluster-tls-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580868 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580876 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8160ae4c-99ef-4e49-af91-403cfa393667-config-out\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580884 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-web-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.580938 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.580891 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8160ae4c-99ef-4e49-af91-403cfa393667-config-volume\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:19.953526 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953492 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00" exitCode=0 Sep 17 20:52:19.953526 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953518 2579 generic.go:356] "Generic (PLEG): container finished" podID="8160ae4c-99ef-4e49-af91-403cfa393667" containerID="51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f" exitCode=0 Sep 17 20:52:19.953756 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00"} Sep 17 20:52:19.953756 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953604 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:19.953756 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f"} Sep 17 20:52:19.953756 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8160ae4c-99ef-4e49-af91-403cfa393667","Type":"ContainerDied","Data":"b8b48cecf81257e6818769888b9995487acf3d4842d4e1facef27b89e279d269"} Sep 17 20:52:19.953756 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.953747 2579 scope.go:117] "RemoveContainer" containerID="ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3" Sep 17 20:52:19.961976 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.961954 2579 scope.go:117] "RemoveContainer" containerID="8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00" Sep 17 20:52:19.969852 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.969836 2579 scope.go:117] "RemoveContainer" containerID="3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110" Sep 17 20:52:19.976687 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.976670 2579 scope.go:117] "RemoveContainer" containerID="51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f" Sep 17 20:52:19.983412 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.983386 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:52:19.983996 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.983976 2579 scope.go:117] "RemoveContainer" containerID="f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f" Sep 17 20:52:19.991421 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.991399 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:52:19.991843 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.991826 2579 scope.go:117] "RemoveContainer" containerID="68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58" Sep 17 20:52:19.998735 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:19.998718 2579 scope.go:117] "RemoveContainer" containerID="8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22" Sep 17 20:52:20.005450 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.005435 2579 scope.go:117] "RemoveContainer" containerID="ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3" Sep 17 20:52:20.005731 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.005713 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3\": container with ID starting with ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3 not found: ID does not exist" containerID="ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3" Sep 17 20:52:20.005796 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.005743 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3"} err="failed to get container status \"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3\": rpc error: code = NotFound desc = could not find container \"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3\": container with ID starting with ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3 not found: ID does not exist" Sep 17 20:52:20.005796 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.005767 2579 scope.go:117] "RemoveContainer" containerID="8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00" Sep 17 20:52:20.006042 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.006026 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00\": container with ID starting with 8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00 not found: ID does not exist" containerID="8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00" Sep 17 20:52:20.006077 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006053 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00"} err="failed to get container status \"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00\": rpc error: code = NotFound desc = could not find container \"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00\": container with ID starting with 8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00 not found: ID does not exist" Sep 17 20:52:20.006077 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006069 2579 scope.go:117] "RemoveContainer" containerID="3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110" Sep 17 20:52:20.006309 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.006293 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110\": container with ID starting with 3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110 not found: ID does not exist" containerID="3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110" Sep 17 20:52:20.006354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006314 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110"} err="failed to get container status \"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110\": rpc error: code = NotFound desc = could not find container \"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110\": container with ID starting with 3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110 not found: ID does not exist" Sep 17 20:52:20.006354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006330 2579 scope.go:117] "RemoveContainer" containerID="51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f" Sep 17 20:52:20.006578 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.006559 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f\": container with ID starting with 51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f not found: ID does not exist" containerID="51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f" Sep 17 20:52:20.006647 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006583 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f"} err="failed to get container status \"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f\": rpc error: code = NotFound desc = could not find container \"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f\": container with ID starting with 51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f not found: ID does not exist" Sep 17 20:52:20.006647 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006602 2579 scope.go:117] "RemoveContainer" containerID="f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f" Sep 17 20:52:20.006866 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.006849 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f\": container with ID starting with f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f not found: ID does not exist" containerID="f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f" Sep 17 20:52:20.006904 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006870 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f"} err="failed to get container status \"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f\": rpc error: code = NotFound desc = could not find container \"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f\": container with ID starting with f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f not found: ID does not exist" Sep 17 20:52:20.006904 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.006888 2579 scope.go:117] "RemoveContainer" containerID="68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58" Sep 17 20:52:20.007115 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.007093 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58\": container with ID starting with 68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58 not found: ID does not exist" containerID="68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58" Sep 17 20:52:20.007204 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007117 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58"} err="failed to get container status \"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58\": rpc error: code = NotFound desc = could not find container \"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58\": container with ID starting with 68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58 not found: ID does not exist" Sep 17 20:52:20.007204 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007131 2579 scope.go:117] "RemoveContainer" containerID="8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22" Sep 17 20:52:20.007329 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:20.007315 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22\": container with ID starting with 8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22 not found: ID does not exist" containerID="8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22" Sep 17 20:52:20.007396 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007332 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22"} err="failed to get container status \"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22\": rpc error: code = NotFound desc = could not find container \"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22\": container with ID starting with 8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22 not found: ID does not exist" Sep 17 20:52:20.007396 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007346 2579 scope.go:117] "RemoveContainer" containerID="ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3" Sep 17 20:52:20.007582 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007565 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3"} err="failed to get container status \"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3\": rpc error: code = NotFound desc = could not find container \"ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3\": container with ID starting with ca329ec2f3f99bec3dc24781a6aab16b9f2e9d14252cdcc1d32b2a9ba8556ac3 not found: ID does not exist" Sep 17 20:52:20.007631 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007584 2579 scope.go:117] "RemoveContainer" containerID="8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00" Sep 17 20:52:20.007810 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007792 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00"} err="failed to get container status \"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00\": rpc error: code = NotFound desc = could not find container \"8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00\": container with ID starting with 8de11a54f5163a1d2f9bf00b5de5eb550091365fc3833d206935e0dc3574ed00 not found: ID does not exist" Sep 17 20:52:20.007879 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.007812 2579 scope.go:117] "RemoveContainer" containerID="3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110" Sep 17 20:52:20.008049 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008035 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110"} err="failed to get container status \"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110\": rpc error: code = NotFound desc = could not find container \"3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110\": container with ID starting with 3a1dcc7874a72a5560a77c70f6fb8e65fdaec01c48aba7023417998c62983110 not found: ID does not exist" Sep 17 20:52:20.008151 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008050 2579 scope.go:117] "RemoveContainer" containerID="51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f" Sep 17 20:52:20.008259 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008243 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f"} err="failed to get container status \"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f\": rpc error: code = NotFound desc = could not find container \"51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f\": container with ID starting with 51659b88cc3945c0748d092e2bbafaedadf04ce1b756f1603b9033d6f740395f not found: ID does not exist" Sep 17 20:52:20.008259 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008258 2579 scope.go:117] "RemoveContainer" containerID="f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f" Sep 17 20:52:20.008509 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008489 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f"} err="failed to get container status \"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f\": rpc error: code = NotFound desc = could not find container \"f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f\": container with ID starting with f42252b154d360a498db8661888573a2cb53eb1e0601e92f3393d3e1efdc8d6f not found: ID does not exist" Sep 17 20:52:20.008581 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008509 2579 scope.go:117] "RemoveContainer" containerID="68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58" Sep 17 20:52:20.008723 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008707 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58"} err="failed to get container status \"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58\": rpc error: code = NotFound desc = could not find container \"68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58\": container with ID starting with 68fd6ac6df73c434880276d0dfe4422c6bbf9cad34168b837a77be2cddab5e58 not found: ID does not exist" Sep 17 20:52:20.008789 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008726 2579 scope.go:117] "RemoveContainer" containerID="8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22" Sep 17 20:52:20.009009 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.008992 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22"} err="failed to get container status \"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22\": rpc error: code = NotFound desc = could not find container \"8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22\": container with ID starting with 8e8d4c1fefe26c3f924f7cbbd12a33f7310353a17789688564352b7a9ddb0c22 not found: ID does not exist" Sep 17 20:52:20.034420 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034397 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:52:20.034666 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034655 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="prom-label-proxy" Sep 17 20:52:20.034708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034669 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="prom-label-proxy" Sep 17 20:52:20.034708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034678 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bedbe960-e01b-43b8-8e80-f20abee6c1a3" containerName="console" Sep 17 20:52:20.034708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034683 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bedbe960-e01b-43b8-8e80-f20abee6c1a3" containerName="console" Sep 17 20:52:20.034708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034692 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="init-config-reloader" Sep 17 20:52:20.034708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034698 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="init-config-reloader" Sep 17 20:52:20.034708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034706 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="config-reloader" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034711 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="config-reloader" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034723 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034731 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d974800-a495-41e7-87e6-69e28de6156f" containerName="registry" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034736 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d974800-a495-41e7-87e6-69e28de6156f" containerName="registry" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034741 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d215d4c-d81f-440c-a3f7-d153cdecdd5b" containerName="console" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034747 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d215d4c-d81f-440c-a3f7-d153cdecdd5b" containerName="console" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034752 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="alertmanager" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034757 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="alertmanager" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034763 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-metric" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034768 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-metric" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034776 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-web" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034781 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-web" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034850 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-metric" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034858 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d215d4c-d81f-440c-a3f7-d153cdecdd5b" containerName="console" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034864 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034871 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d974800-a495-41e7-87e6-69e28de6156f" containerName="registry" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034875 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="kube-rbac-proxy-web" Sep 17 20:52:20.034877 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034880 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="prom-label-proxy" Sep 17 20:52:20.035459 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034885 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="config-reloader" Sep 17 20:52:20.035459 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034890 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" containerName="alertmanager" Sep 17 20:52:20.035459 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.034895 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bedbe960-e01b-43b8-8e80-f20abee6c1a3" containerName="console" Sep 17 20:52:20.038068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.038054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.042854 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.042816 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 20:52:20.042933 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.042850 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 20:52:20.043105 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 20:52:20.043164 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043142 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 20:52:20.043164 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gdt2x\"" Sep 17 20:52:20.043260 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043171 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 20:52:20.043260 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043224 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 20:52:20.043260 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043235 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 20:52:20.043637 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.043622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 20:52:20.049041 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.049024 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 20:52:20.058915 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.058896 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:52:20.083188 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083278 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083278 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083278 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083278 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-config-out\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083536 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-web-config\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083536 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083536 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083536 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.083657 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.083539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkr78\" (UniqueName: \"kubernetes.io/projected/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-kube-api-access-hkr78\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183952 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-config-out\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183952 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183952 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183952 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.183952 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-web-config\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.184165 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.184165 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.183980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.184165 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.184012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.184417 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.184391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.184959 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.184934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.185057 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.184995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkr78\" (UniqueName: \"kubernetes.io/projected/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-kube-api-access-hkr78\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.185057 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.185045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.185163 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.185073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.185290 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.185263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.187195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.187117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.187294 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.187231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-config-out\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.187656 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.187637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.188264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.188139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.188264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.188152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-web-config\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.188264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.188188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.188264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.188195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.188264 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.188192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.188890 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.188870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.196417 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.196396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkr78\" (UniqueName: \"kubernetes.io/projected/ea96a6e2-af38-4510-8d3d-fc2353f9a78d-kube-api-access-hkr78\") pod \"alertmanager-main-0\" (UID: \"ea96a6e2-af38-4510-8d3d-fc2353f9a78d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.348435 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.348414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 20:52:20.644844 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.644817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 20:52:20.646017 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:52:20.645992 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea96a6e2_af38_4510_8d3d_fc2353f9a78d.slice/crio-586a1525ab03237a0b2de97c1834f64ef2bdc985b56bec2efab23e37315e76ee WatchSource:0}: Error finding container 586a1525ab03237a0b2de97c1834f64ef2bdc985b56bec2efab23e37315e76ee: Status 404 returned error can't find the container with id 586a1525ab03237a0b2de97c1834f64ef2bdc985b56bec2efab23e37315e76ee Sep 17 20:52:20.965020 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.964958 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea96a6e2-af38-4510-8d3d-fc2353f9a78d" containerID="0bdbbf4c279ce9e53c619a640730519fb3b14e082feb1515d98c8b0c0c2bb527" exitCode=0 Sep 17 20:52:20.965147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.965044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerDied","Data":"0bdbbf4c279ce9e53c619a640730519fb3b14e082feb1515d98c8b0c0c2bb527"} Sep 17 20:52:20.965202 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:20.965143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"586a1525ab03237a0b2de97c1834f64ef2bdc985b56bec2efab23e37315e76ee"} Sep 17 20:52:21.053990 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.053967 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8160ae4c-99ef-4e49-af91-403cfa393667" path="/var/lib/kubelet/pods/8160ae4c-99ef-4e49-af91-403cfa393667/volumes" Sep 17 20:52:21.976527 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.976493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"f3d945a9ed9bb555d190575f10c8a438585b7017bcb3af1ce808072e7baf2940"} Sep 17 20:52:21.976912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.976560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"33848c6176798df00687e56c1ad9033c4f4c44585f0fd2f4e30b891890ff07e3"} Sep 17 20:52:21.976912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.976587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"206ffd3f730ee7593971b930dbe2a9109dcdbe09cfabf40579785bc4dc7d8321"} Sep 17 20:52:21.976912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.976613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"887a124df384a03dada061109f3f565644f18dcaddb7ab6849ec6da432234e88"} Sep 17 20:52:21.976912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.976637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"1bf0ea95916be4f30e704f13997bbfab98c7211f8308767767d323da6bcd1f6a"} Sep 17 20:52:21.976912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:21.976662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea96a6e2-af38-4510-8d3d-fc2353f9a78d","Type":"ContainerStarted","Data":"90a50b996b3e481a69019404453cb253f2a25505fb3a1e0caf55a7151a50fad5"} Sep 17 20:52:22.062912 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.062865 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.047587848 podStartE2EDuration="2.062850708s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.26286ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:22.061533985 +0000 UTC m=+187.536216328" watchObservedRunningTime="2026-09-17 20:52:22.062850708 +0000 UTC m=+187.537533059" Sep 17 20:52:22.415415 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415356 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:52:22.415794 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415710 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="prometheus" containerID="cri-o://57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" gracePeriod=600 Sep 17 20:52:22.415794 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415747 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" gracePeriod=600 Sep 17 20:52:22.415794 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415743 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-web" containerID="cri-o://6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" gracePeriod=600 Sep 17 20:52:22.415995 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415742 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="thanos-sidecar" containerID="cri-o://0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" gracePeriod=600 Sep 17 20:52:22.415995 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415767 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy" containerID="cri-o://04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" gracePeriod=600 Sep 17 20:52:22.415995 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.415762 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="config-reloader" containerID="cri-o://124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" gracePeriod=600 Sep 17 20:52:22.649621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.649599 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:22.705199 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705153 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-db\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705199 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-web-config\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705216 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-trusted-ca-bundle\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-grpc-tls\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705312 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-kubelet-serving-ca-bundle\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705354 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705348 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-thanos-prometheus-http-client-file\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705394 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcp2v\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-kube-api-access-gcp2v\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705424 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-2\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705452 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-kube-rbac-proxy\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-1\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705501 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-tls-assets\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705538 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-config-out\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-serving-certs-ca-bundle\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705587 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-tls\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-config\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705631 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-metrics-client-ca\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.705681 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-0\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.706290 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705704 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.706290 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705762 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-metrics-client-certs\") pod \"33490c30-d618-4837-a9b5-28708600092d\" (UID: \"33490c30-d618-4837-a9b5-28708600092d\") " Sep 17 20:52:22.706290 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705981 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.706290 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.705976 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.706731 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.706298 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.706731 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.706561 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 20:52:22.706919 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.706853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.707388 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.707336 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.708784 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.708755 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.709178 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.709153 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 20:52:22.710563 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.710527 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-kube-api-access-gcp2v" (OuterVolumeSpecName: "kube-api-access-gcp2v") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "kube-api-access-gcp2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:52:22.710988 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.710803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-config-out" (OuterVolumeSpecName: "config-out") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 20:52:22.710988 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.710820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.710988 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.710842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.710988 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.710907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.711330 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.711054 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.711637 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.711600 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.711837 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.711814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.711922 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.711836 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-config" (OuterVolumeSpecName: "config") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.712738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.712697 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.712852 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.712835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:52:22.721604 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.721577 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-web-config" (OuterVolumeSpecName: "web-config") pod "33490c30-d618-4837-a9b5-28708600092d" (UID: "33490c30-d618-4837-a9b5-28708600092d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 20:52:22.807042 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807020 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807042 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807042 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807053 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807062 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcp2v\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-kube-api-access-gcp2v\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807070 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807080 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-kube-rbac-proxy\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807088 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807097 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33490c30-d618-4837-a9b5-28708600092d-tls-assets\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807105 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-config-out\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807112 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807120 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807128 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807135 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-configmap-metrics-client-ca\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807142 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807151 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807159 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-metrics-client-certs\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807166 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33490c30-d618-4837-a9b5-28708600092d-prometheus-k8s-db\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807173 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-web-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.807194 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.807181 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33490c30-d618-4837-a9b5-28708600092d-secret-grpc-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:52:22.988967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988898 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" exitCode=0 Sep 17 20:52:22.988967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988920 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" exitCode=0 Sep 17 20:52:22.988967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988927 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" exitCode=0 Sep 17 20:52:22.988967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988933 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" exitCode=0 Sep 17 20:52:22.988967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988938 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" exitCode=0 Sep 17 20:52:22.988967 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988943 2579 generic.go:356] "Generic (PLEG): container finished" podID="33490c30-d618-4837-a9b5-28708600092d" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" exitCode=0 Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.988991 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989068 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989194 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} Sep 17 20:52:22.989605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.989352 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33490c30-d618-4837-a9b5-28708600092d","Type":"ContainerDied","Data":"f0564e6d0d8188301c623849581d92ae8550d2e0a99f444b90287fa9405f2a3e"} Sep 17 20:52:22.998215 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:22.998026 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.005646 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.005631 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.012651 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.012630 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.019148 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.019111 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.020886 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.020867 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:52:23.027158 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.027144 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.027705 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.027650 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:52:23.034103 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.034089 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.040390 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.040358 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.040650 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.040631 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.040703 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.040656 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} err="failed to get container status \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" Sep 17 20:52:23.040703 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.040673 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.040918 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.040903 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.040959 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.040922 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} err="failed to get container status \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" Sep 17 20:52:23.040959 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.040934 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.041149 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.041134 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.041190 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041154 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} err="failed to get container status \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" Sep 17 20:52:23.041190 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041168 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.041379 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.041352 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.041426 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041398 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} err="failed to get container status \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" Sep 17 20:52:23.041426 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041410 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.041648 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.041633 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.041701 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041648 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} err="failed to get container status \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" Sep 17 20:52:23.041701 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041659 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.041880 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.041864 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.041924 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041883 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} err="failed to get container status \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" Sep 17 20:52:23.041924 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.041895 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.042125 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:52:23.042108 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.042167 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042129 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} err="failed to get container status \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" Sep 17 20:52:23.042167 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042142 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.042352 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042336 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} err="failed to get container status \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" Sep 17 20:52:23.042419 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042353 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.042573 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042557 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} err="failed to get container status \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" Sep 17 20:52:23.042613 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042575 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.042759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042744 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} err="failed to get container status \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" Sep 17 20:52:23.042806 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042759 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.042951 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042934 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} err="failed to get container status \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" Sep 17 20:52:23.042992 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.042951 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.043173 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043157 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} err="failed to get container status \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" Sep 17 20:52:23.043214 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043173 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.043390 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043361 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} err="failed to get container status \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" Sep 17 20:52:23.043430 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043390 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.043596 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043581 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} err="failed to get container status \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" Sep 17 20:52:23.043638 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043596 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.043830 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043814 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} err="failed to get container status \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" Sep 17 20:52:23.043871 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043829 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.044000 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043985 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} err="failed to get container status \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" Sep 17 20:52:23.044000 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.043999 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.044195 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044182 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} err="failed to get container status \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" Sep 17 20:52:23.044234 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044195 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.044415 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044398 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} err="failed to get container status \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" Sep 17 20:52:23.044474 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044418 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.044640 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044624 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} err="failed to get container status \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" Sep 17 20:52:23.044679 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044640 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.044833 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044816 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} err="failed to get container status \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" Sep 17 20:52:23.044881 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.044833 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.045056 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045041 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} err="failed to get container status \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" Sep 17 20:52:23.045091 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045056 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.045237 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045222 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} err="failed to get container status \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" Sep 17 20:52:23.045272 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045237 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.045439 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045421 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} err="failed to get container status \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" Sep 17 20:52:23.045493 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045440 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.045627 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045611 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} err="failed to get container status \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" Sep 17 20:52:23.045673 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045628 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.045816 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} err="failed to get container status \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" Sep 17 20:52:23.045859 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045817 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.045980 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045965 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} err="failed to get container status \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" Sep 17 20:52:23.046029 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.045980 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.046161 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046142 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} err="failed to get container status \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" Sep 17 20:52:23.046225 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046163 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.046356 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046342 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} err="failed to get container status \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" Sep 17 20:52:23.046356 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046355 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.046558 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046541 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} err="failed to get container status \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" Sep 17 20:52:23.046599 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046559 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.046757 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046740 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} err="failed to get container status \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" Sep 17 20:52:23.046830 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046759 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.046955 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046940 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} err="failed to get container status \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" Sep 17 20:52:23.046997 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.046955 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.047147 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047129 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} err="failed to get container status \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" Sep 17 20:52:23.047216 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047148 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.047412 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047393 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} err="failed to get container status \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" Sep 17 20:52:23.047412 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047410 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.047640 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047616 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} err="failed to get container status \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" Sep 17 20:52:23.047694 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047643 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.047915 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047899 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} err="failed to get container status \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" Sep 17 20:52:23.047972 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.047915 2579 scope.go:117] "RemoveContainer" containerID="d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9" Sep 17 20:52:23.048135 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048119 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9"} err="failed to get container status \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": rpc error: code = NotFound desc = could not find container \"d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9\": container with ID starting with d100ded82b4e879cd9b6ddc9e4c7beca1e766f49e5f12e45a2111272302dadb9 not found: ID does not exist" Sep 17 20:52:23.048135 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048134 2579 scope.go:117] "RemoveContainer" containerID="04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07" Sep 17 20:52:23.048382 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048351 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07"} err="failed to get container status \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": rpc error: code = NotFound desc = could not find container \"04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07\": container with ID starting with 04d3f8a1563f16a704f9ee64599e3c3b8401751f6fb94f1b25b8e8796eaf9c07 not found: ID does not exist" Sep 17 20:52:23.048454 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048387 2579 scope.go:117] "RemoveContainer" containerID="6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813" Sep 17 20:52:23.048620 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048604 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813"} err="failed to get container status \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": rpc error: code = NotFound desc = could not find container \"6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813\": container with ID starting with 6b0c8aad5ca978dd4e5f2327e276e018c1557e70f3ec8523f4f52de39593d813 not found: ID does not exist" Sep 17 20:52:23.048620 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048621 2579 scope.go:117] "RemoveContainer" containerID="0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0" Sep 17 20:52:23.048862 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048847 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0"} err="failed to get container status \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": rpc error: code = NotFound desc = could not find container \"0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0\": container with ID starting with 0c714ce126bf5f4da46128caaf6e9c6c64a589c24c19210bf8e4b1f9f1fd83f0 not found: ID does not exist" Sep 17 20:52:23.048904 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.048862 2579 scope.go:117] "RemoveContainer" containerID="124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b" Sep 17 20:52:23.049063 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.049046 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b"} err="failed to get container status \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": rpc error: code = NotFound desc = could not find container \"124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b\": container with ID starting with 124bcc276b3abac14c58e1c617af3e727125e387d3f70c3f1292e449f0037c8b not found: ID does not exist" Sep 17 20:52:23.049063 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.049062 2579 scope.go:117] "RemoveContainer" containerID="57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985" Sep 17 20:52:23.049265 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.049250 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985"} err="failed to get container status \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": rpc error: code = NotFound desc = could not find container \"57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985\": container with ID starting with 57d05e694d1951a00ae89928e07fd32507c6399ceeb23c4bc84751961d7d2985 not found: ID does not exist" Sep 17 20:52:23.049308 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.049265 2579 scope.go:117] "RemoveContainer" containerID="aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c" Sep 17 20:52:23.049524 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.049507 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c"} err="failed to get container status \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": rpc error: code = NotFound desc = could not find container \"aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c\": container with ID starting with aa85f05e27a7ad2085e7c376e4b571541d78d655d44d3bc5066932fe0e3f970c not found: ID does not exist" Sep 17 20:52:23.052630 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.052613 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33490c30-d618-4837-a9b5-28708600092d" path="/var/lib/kubelet/pods/33490c30-d618-4837-a9b5-28708600092d/volumes" Sep 17 20:52:23.074170 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074147 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:52:23.074461 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074449 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-web" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074464 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-web" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074474 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="init-config-reloader" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074481 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="init-config-reloader" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074496 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="prometheus" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074502 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="prometheus" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074511 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-thanos" Sep 17 20:52:23.074515 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074516 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-thanos" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074527 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="config-reloader" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074533 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="config-reloader" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074539 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074543 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074549 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="thanos-sidecar" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074554 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="thanos-sidecar" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074607 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="config-reloader" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074615 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-web" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074622 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074629 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="prometheus" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074634 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="thanos-sidecar" Sep 17 20:52:23.074708 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.074639 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33490c30-d618-4837-a9b5-28708600092d" containerName="kube-rbac-proxy-thanos" Sep 17 20:52:23.079794 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.079780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.090703 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.090687 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 20:52:23.090703 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.090699 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 20:52:23.093296 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 20:52:23.093360 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093318 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 20:52:23.093360 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 20:52:23.093492 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093414 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 20:52:23.093492 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093415 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 20:52:23.093492 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093463 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 20:52:23.093620 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093566 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 20:52:23.093693 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093675 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5pvrq\"" Sep 17 20:52:23.093791 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093775 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 20:52:23.093866 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093852 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 20:52:23.093993 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.093979 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 20:52:23.094227 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.094210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 20:52:23.094358 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.094345 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-57f9qld6qn70h\"" Sep 17 20:52:23.096079 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.096063 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 20:52:23.099460 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.099442 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 20:52:23.109905 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.109880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9bx\" (UniqueName: \"kubernetes.io/projected/fff22ab4-3ac9-462e-92aa-83931293e56a-kube-api-access-6z9bx\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.109983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.109910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.109983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.109926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.109983 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.109959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110096 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110131 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110161 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110236 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110236 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110293 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-web-config\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110328 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110361 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110444 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110480 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-config\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110480 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fff22ab4-3ac9-462e-92aa-83931293e56a-config-out\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110546 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110546 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110608 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fff22ab4-3ac9-462e-92aa-83931293e56a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110608 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.110608 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.110601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.118118 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.118099 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:52:23.211351 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211455 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211455 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211531 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211568 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211623 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-web-config\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211623 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211738 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211778 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211828 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-config\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211828 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fff22ab4-3ac9-462e-92aa-83931293e56a-config-out\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211925 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211925 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.211925 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fff22ab4-3ac9-462e-92aa-83931293e56a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212061 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212061 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212061 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.211978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9bx\" (UniqueName: \"kubernetes.io/projected/fff22ab4-3ac9-462e-92aa-83931293e56a-kube-api-access-6z9bx\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212061 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.212004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212061 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.212017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212061 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.212031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.212611 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.212082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.214748 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.213464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.214748 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.214315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.214748 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.214585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.214961 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.214925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215293 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215404 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215542 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215617 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215781 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215843 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.215884 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fff22ab4-3ac9-462e-92aa-83931293e56a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.216020 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.215998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.216710 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.216679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.216837 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.216814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-web-config\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.216924 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.216907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-config\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.217388 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.217341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fff22ab4-3ac9-462e-92aa-83931293e56a-config-out\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.217746 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.217728 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fff22ab4-3ac9-462e-92aa-83931293e56a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.217977 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.217961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fff22ab4-3ac9-462e-92aa-83931293e56a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.227964 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.227941 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9bx\" (UniqueName: \"kubernetes.io/projected/fff22ab4-3ac9-462e-92aa-83931293e56a-kube-api-access-6z9bx\") pod \"prometheus-k8s-0\" (UID: \"fff22ab4-3ac9-462e-92aa-83931293e56a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.388920 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.388887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:52:23.714700 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.714678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 20:52:23.716809 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:52:23.716786 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff22ab4_3ac9_462e_92aa_83931293e56a.slice/crio-f835752c9561bc15f57108b50b07f9282a871ea52ac7f0c37f45e34d76264dd8 WatchSource:0}: Error finding container f835752c9561bc15f57108b50b07f9282a871ea52ac7f0c37f45e34d76264dd8: Status 404 returned error can't find the container with id f835752c9561bc15f57108b50b07f9282a871ea52ac7f0c37f45e34d76264dd8 Sep 17 20:52:23.994605 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.994535 2579 generic.go:356] "Generic (PLEG): container finished" podID="fff22ab4-3ac9-462e-92aa-83931293e56a" containerID="af55326edeb0f542ef2d1a10e210892d36d72a3f8a843f84de0a91c2d6a33ea3" exitCode=0 Sep 17 20:52:23.994953 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.994622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerDied","Data":"af55326edeb0f542ef2d1a10e210892d36d72a3f8a843f84de0a91c2d6a33ea3"} Sep 17 20:52:23.994953 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:23.994697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"f835752c9561bc15f57108b50b07f9282a871ea52ac7f0c37f45e34d76264dd8"} Sep 17 20:52:25.004284 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.004244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"4b6db0dd5c0dc500db9ebcccebd1f4aae897da7dad54317b1a1a954031ab16cf"} Sep 17 20:52:25.004817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.004358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"b007558c5acb63392889302aecaa6f9222f09b0165d22b2bd9f5e001bd68ddfb"} Sep 17 20:52:25.004817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.004425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"7456e51ef179f37c3846a075f45dd16aac42c281fe9eb826e564b99c19e78426"} Sep 17 20:52:25.004817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.004453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"b282e145b1dbbfb0017dc51da1d47ddc8e5c24c713276db5ce205345a19b563d"} Sep 17 20:52:25.004817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.004478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"597bd44c51c68ad13635f68236131dbfab67fae8cf63c2c391d98935dd9b5663"} Sep 17 20:52:25.004817 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.004505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fff22ab4-3ac9-462e-92aa-83931293e56a","Type":"ContainerStarted","Data":"9f17feead7d8c3908641f7e0f8912a04c5fba0d6961d389ec0bd1eb6f4937ee8"} Sep 17 20:52:25.042895 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:25.042842 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.024346019 podStartE2EDuration="2.042822188s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.476169ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:25.041253681 +0000 UTC m=+190.515936080" watchObservedRunningTime="2026-09-17 20:52:25.042822188 +0000 UTC m=+190.517504534" Sep 17 20:52:28.389318 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:52:28.389276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:53:23.389307 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:53:23.389270 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:53:23.404575 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:53:23.404550 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:53:24.256475 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:53:24.256451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 20:54:14.963419 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:54:14.963394 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 20:55:44.436225 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.436191 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-s6sx2"] Sep 17 20:55:44.439460 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.439444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.442544 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.442514 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fzwlq\"" Sep 17 20:55:44.442698 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.442679 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 20:55:44.443642 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.443622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 17 20:55:44.443642 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.443638 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 20:55:44.449939 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.449921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-s6sx2"] Sep 17 20:55:44.555282 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.555259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkgq\" (UniqueName: \"kubernetes.io/projected/519ffa1d-20f5-452b-ac97-4346af6f8255-kube-api-access-clkgq\") pod \"seaweedfs-64568bcd49-s6sx2\" (UID: \"519ffa1d-20f5-452b-ac97-4346af6f8255\") " pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.555431 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.555299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/519ffa1d-20f5-452b-ac97-4346af6f8255-data\") pod \"seaweedfs-64568bcd49-s6sx2\" (UID: \"519ffa1d-20f5-452b-ac97-4346af6f8255\") " pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.656164 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.656134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/519ffa1d-20f5-452b-ac97-4346af6f8255-data\") pod \"seaweedfs-64568bcd49-s6sx2\" (UID: \"519ffa1d-20f5-452b-ac97-4346af6f8255\") " pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.656293 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.656244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkgq\" (UniqueName: \"kubernetes.io/projected/519ffa1d-20f5-452b-ac97-4346af6f8255-kube-api-access-clkgq\") pod \"seaweedfs-64568bcd49-s6sx2\" (UID: \"519ffa1d-20f5-452b-ac97-4346af6f8255\") " pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.656522 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.656500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/519ffa1d-20f5-452b-ac97-4346af6f8255-data\") pod \"seaweedfs-64568bcd49-s6sx2\" (UID: \"519ffa1d-20f5-452b-ac97-4346af6f8255\") " pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.665638 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.665616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkgq\" (UniqueName: \"kubernetes.io/projected/519ffa1d-20f5-452b-ac97-4346af6f8255-kube-api-access-clkgq\") pod \"seaweedfs-64568bcd49-s6sx2\" (UID: \"519ffa1d-20f5-452b-ac97-4346af6f8255\") " pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:44.749199 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:44.749134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:45.023484 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:45.023451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-s6sx2"] Sep 17 20:55:45.026402 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:55:45.026377 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519ffa1d_20f5_452b_ac97_4346af6f8255.slice/crio-2ca63fa96cdf1ea1c0309f803953b03a35fc4611f3d820c6778a0b34333c997f WatchSource:0}: Error finding container 2ca63fa96cdf1ea1c0309f803953b03a35fc4611f3d820c6778a0b34333c997f: Status 404 returned error can't find the container with id 2ca63fa96cdf1ea1c0309f803953b03a35fc4611f3d820c6778a0b34333c997f Sep 17 20:55:45.027676 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:45.027658 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 20:55:45.815254 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:45.815218 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-s6sx2" event={"ID":"519ffa1d-20f5-452b-ac97-4346af6f8255","Type":"ContainerStarted","Data":"2ca63fa96cdf1ea1c0309f803953b03a35fc4611f3d820c6778a0b34333c997f"} Sep 17 20:55:48.830522 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:48.830489 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-s6sx2" event={"ID":"519ffa1d-20f5-452b-ac97-4346af6f8255","Type":"ContainerStarted","Data":"f9b9d335464011f31e741924569d45e44d288e80170f1d223a9be758353d0e7f"} Sep 17 20:55:48.830904 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:48.830635 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:55:48.848069 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:48.847986 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-s6sx2" podStartSLOduration=1.7532344439999998 podStartE2EDuration="4.847965538s" totalImagesPullingTime="3.094731094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:48.846736518 +0000 UTC m=+394.321418863" watchObservedRunningTime="2026-09-17 20:55:48.847965538 +0000 UTC m=+394.322647884" Sep 17 20:55:54.837903 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:55:54.837869 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-s6sx2" Sep 17 20:57:18.140906 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.140863 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8xmb9"] Sep 17 20:57:18.144283 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.144252 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.147602 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.147581 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 17 20:57:18.154703 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.154681 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8xmb9"] Sep 17 20:57:18.316435 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.316396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6753aa2-ef20-4371-9b8b-79db88916230-model-cache\") pod \"s3-init-8xmb9\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.316435 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.316438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whgp5\" (UniqueName: \"kubernetes.io/projected/b6753aa2-ef20-4371-9b8b-79db88916230-kube-api-access-whgp5\") pod \"s3-init-8xmb9\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.417803 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.417735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6753aa2-ef20-4371-9b8b-79db88916230-model-cache\") pod \"s3-init-8xmb9\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.417803 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.417770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whgp5\" (UniqueName: \"kubernetes.io/projected/b6753aa2-ef20-4371-9b8b-79db88916230-kube-api-access-whgp5\") pod \"s3-init-8xmb9\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.418068 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.418048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6753aa2-ef20-4371-9b8b-79db88916230-model-cache\") pod \"s3-init-8xmb9\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.428603 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.428585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whgp5\" (UniqueName: \"kubernetes.io/projected/b6753aa2-ef20-4371-9b8b-79db88916230-kube-api-access-whgp5\") pod \"s3-init-8xmb9\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.453976 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.453955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8xmb9" Sep 17 20:57:18.726828 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:18.726798 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8xmb9"] Sep 17 20:57:18.728006 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:57:18.727982 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6753aa2_ef20_4371_9b8b_79db88916230.slice/crio-205927edea93d562914323e9f47b7446d365de47747cb5613bd2ad1bc2f528ae WatchSource:0}: Error finding container 205927edea93d562914323e9f47b7446d365de47747cb5613bd2ad1bc2f528ae: Status 404 returned error can't find the container with id 205927edea93d562914323e9f47b7446d365de47747cb5613bd2ad1bc2f528ae Sep 17 20:57:19.185665 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:19.185633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8xmb9" event={"ID":"b6753aa2-ef20-4371-9b8b-79db88916230","Type":"ContainerStarted","Data":"205927edea93d562914323e9f47b7446d365de47747cb5613bd2ad1bc2f528ae"} Sep 17 20:57:23.204733 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:23.204691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8xmb9" event={"ID":"b6753aa2-ef20-4371-9b8b-79db88916230","Type":"ContainerStarted","Data":"2f413e7a194f1c170a162dab1813d9633da5f57c8816aa5bc8043bd3c5c3be3f"} Sep 17 20:57:24.210157 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:24.210124 2579 generic.go:356] "Generic (PLEG): container finished" podID="b6753aa2-ef20-4371-9b8b-79db88916230" containerID="2f413e7a194f1c170a162dab1813d9633da5f57c8816aa5bc8043bd3c5c3be3f" exitCode=0 Sep 17 20:57:24.210550 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:24.210187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8xmb9" event={"ID":"b6753aa2-ef20-4371-9b8b-79db88916230","Type":"ContainerDied","Data":"2f413e7a194f1c170a162dab1813d9633da5f57c8816aa5bc8043bd3c5c3be3f"} Sep 17 20:57:29.235076 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:57:29.235041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8xmb9" event={"ID":"b6753aa2-ef20-4371-9b8b-79db88916230","Type":"ContainerStarted","Data":"bb5f5ead8761c4c05d42ca77154a8b560f3f14176e63f8fe31c7679c9131aea7"} Sep 17 20:58:33.507157 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:33.507127 2579 generic.go:356] "Generic (PLEG): container finished" podID="b6753aa2-ef20-4371-9b8b-79db88916230" containerID="bb5f5ead8761c4c05d42ca77154a8b560f3f14176e63f8fe31c7679c9131aea7" exitCode=0 Sep 17 20:58:33.507534 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:33.507200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8xmb9" event={"ID":"b6753aa2-ef20-4371-9b8b-79db88916230","Type":"ContainerDied","Data":"bb5f5ead8761c4c05d42ca77154a8b560f3f14176e63f8fe31c7679c9131aea7"} Sep 17 20:58:34.633753 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.633730 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8xmb9" Sep 17 20:58:34.781326 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.781269 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6753aa2-ef20-4371-9b8b-79db88916230-model-cache\") pod \"b6753aa2-ef20-4371-9b8b-79db88916230\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " Sep 17 20:58:34.781462 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.781326 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whgp5\" (UniqueName: \"kubernetes.io/projected/b6753aa2-ef20-4371-9b8b-79db88916230-kube-api-access-whgp5\") pod \"b6753aa2-ef20-4371-9b8b-79db88916230\" (UID: \"b6753aa2-ef20-4371-9b8b-79db88916230\") " Sep 17 20:58:34.783624 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.783599 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6753aa2-ef20-4371-9b8b-79db88916230-kube-api-access-whgp5" (OuterVolumeSpecName: "kube-api-access-whgp5") pod "b6753aa2-ef20-4371-9b8b-79db88916230" (UID: "b6753aa2-ef20-4371-9b8b-79db88916230"). InnerVolumeSpecName "kube-api-access-whgp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 20:58:34.799926 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.799905 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6753aa2-ef20-4371-9b8b-79db88916230-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6753aa2-ef20-4371-9b8b-79db88916230" (UID: "b6753aa2-ef20-4371-9b8b-79db88916230"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 20:58:34.882729 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.882709 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6753aa2-ef20-4371-9b8b-79db88916230-model-cache\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:58:34.882729 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:34.882730 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whgp5\" (UniqueName: \"kubernetes.io/projected/b6753aa2-ef20-4371-9b8b-79db88916230-kube-api-access-whgp5\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 20:58:35.516643 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:35.516607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8xmb9" event={"ID":"b6753aa2-ef20-4371-9b8b-79db88916230","Type":"ContainerDied","Data":"205927edea93d562914323e9f47b7446d365de47747cb5613bd2ad1bc2f528ae"} Sep 17 20:58:35.516643 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:35.516631 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8xmb9" Sep 17 20:58:35.516643 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:35.516645 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="205927edea93d562914323e9f47b7446d365de47747cb5613bd2ad1bc2f528ae" Sep 17 20:58:48.352958 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.352926 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps"] Sep 17 20:58:48.353333 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.353258 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6753aa2-ef20-4371-9b8b-79db88916230" containerName="s3-init" Sep 17 20:58:48.353333 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.353269 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6753aa2-ef20-4371-9b8b-79db88916230" containerName="s3-init" Sep 17 20:58:48.353333 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.353282 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6753aa2-ef20-4371-9b8b-79db88916230" containerName="download-hf-model" Sep 17 20:58:48.353333 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.353287 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6753aa2-ef20-4371-9b8b-79db88916230" containerName="download-hf-model" Sep 17 20:58:48.353646 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.353336 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6753aa2-ef20-4371-9b8b-79db88916230" containerName="s3-init" Sep 17 20:58:48.356563 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.356545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.360619 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.360595 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 20:58:48.360749 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.360641 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" Sep 17 20:58:48.360990 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.360971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" Sep 17 20:58:48.361027 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.361004 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 20:58:48.361522 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.361507 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 20:58:48.361583 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.361566 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx4r4\"" Sep 17 20:58:48.361649 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.361634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 20:58:48.376621 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.376600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps"] Sep 17 20:58:48.475700 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.475671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrp6\" (UniqueName: \"kubernetes.io/projected/7534274d-cffc-4e7c-b216-b7141ee17518-kube-api-access-cxrp6\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.475838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.475714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.475838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.475750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.475838 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.475815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.475993 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.475871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7534274d-cffc-4e7c-b216-b7141ee17518-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.576867 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.576841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrp6\" (UniqueName: \"kubernetes.io/projected/7534274d-cffc-4e7c-b216-b7141ee17518-kube-api-access-cxrp6\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.576994 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.576882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.576994 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.576908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.576994 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.576936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.576994 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.576977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7534274d-cffc-4e7c-b216-b7141ee17518-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.577202 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:58:48.577187 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Sep 17 20:58:48.577291 ip-10-0-140-118 kubenswrapper[2579]: E0917 20:58:48.577268 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls podName:7534274d-cffc-4e7c-b216-b7141ee17518 nodeName:}" failed. No retries permitted until 2026-09-17 20:58:49.077245243 +0000 UTC m=+574.551927580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls") pod "isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" (UID: "7534274d-cffc-4e7c-b216-b7141ee17518") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Sep 17 20:58:48.577470 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.577443 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7534274d-cffc-4e7c-b216-b7141ee17518-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.577640 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.577619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.577744 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.577723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.588201 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.588170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrp6\" (UniqueName: \"kubernetes.io/projected/7534274d-cffc-4e7c-b216-b7141ee17518-kube-api-access-cxrp6\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:48.806293 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.806263 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5"] Sep 17 20:58:48.809925 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.809909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:48.812884 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.812863 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Sep 17 20:58:48.813496 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.813475 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Sep 17 20:58:48.826385 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.826352 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5"] Sep 17 20:58:48.980485 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.980457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b726354c-87d0-4a8c-9768-1c66f236f316-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:48.980624 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.980508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b726354c-87d0-4a8c-9768-1c66f236f316-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:48.980624 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.980527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttq84\" (UniqueName: \"kubernetes.io/projected/b726354c-87d0-4a8c-9768-1c66f236f316-kube-api-access-ttq84\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:48.980624 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.980546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:48.980624 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:48.980563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.023736 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.023707 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d"] Sep 17 20:58:49.027112 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.027096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.029936 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.029916 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Sep 17 20:58:49.030237 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.030222 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Sep 17 20:58:49.053906 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.053879 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d"] Sep 17 20:58:49.081325 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:49.081325 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b726354c-87d0-4a8c-9768-1c66f236f316-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.081458 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b726354c-87d0-4a8c-9768-1c66f236f316-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.081458 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttq84\" (UniqueName: \"kubernetes.io/projected/b726354c-87d0-4a8c-9768-1c66f236f316-kube-api-access-ttq84\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.081458 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.081571 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.081808 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.081783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b726354c-87d0-4a8c-9768-1c66f236f316-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.082107 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.082085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.082201 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.082108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.084024 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.084004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:49.084154 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.084076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b726354c-87d0-4a8c-9768-1c66f236f316-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.096438 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.096417 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttq84\" (UniqueName: \"kubernetes.io/projected/b726354c-87d0-4a8c-9768-1c66f236f316-kube-api-access-ttq84\") pod \"isvc-xgboost-graph-predictor-555fc69d56-f4cj5\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.119894 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.119873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:58:49.182959 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.182897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r2f\" (UniqueName: \"kubernetes.io/projected/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-api-access-66r2f\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.183081 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.182969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.183149 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.183092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.183149 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.183128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf379d2-56bc-4856-96cc-3bfba4ea51fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.183267 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.183162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.267232 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.267202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:58:49.284410 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.284383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.284513 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.284439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r2f\" (UniqueName: \"kubernetes.io/projected/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-api-access-66r2f\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.284513 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.284465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.284622 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.284579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.284622 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.284611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf379d2-56bc-4856-96cc-3bfba4ea51fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.284791 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.284719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.285160 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.285139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.285320 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.285298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.286875 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.286846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf379d2-56bc-4856-96cc-3bfba4ea51fc-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.293756 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.293738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r2f\" (UniqueName: \"kubernetes.io/projected/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-api-access-66r2f\") pod \"isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.336966 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.336909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:58:49.396207 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.396177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5"] Sep 17 20:58:49.397970 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:58:49.397941 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb726354c_87d0_4a8c_9768_1c66f236f316.slice/crio-fe1d29001c4ea63287d59b5898b447f8969249ec5878a54afb2f51a0f22deb3b WatchSource:0}: Error finding container fe1d29001c4ea63287d59b5898b447f8969249ec5878a54afb2f51a0f22deb3b: Status 404 returned error can't find the container with id fe1d29001c4ea63287d59b5898b447f8969249ec5878a54afb2f51a0f22deb3b Sep 17 20:58:49.538263 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.538240 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps"] Sep 17 20:58:49.540261 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:58:49.540237 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7534274d_cffc_4e7c_b216_b7141ee17518.slice/crio-d673f775ba129052d5fe236b48cf63d1745fa15486bde65c6bb07ef2d5d3a224 WatchSource:0}: Error finding container d673f775ba129052d5fe236b48cf63d1745fa15486bde65c6bb07ef2d5d3a224: Status 404 returned error can't find the container with id d673f775ba129052d5fe236b48cf63d1745fa15486bde65c6bb07ef2d5d3a224 Sep 17 20:58:49.579936 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.579906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerStarted","Data":"9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a"} Sep 17 20:58:49.580036 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.579974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerStarted","Data":"fe1d29001c4ea63287d59b5898b447f8969249ec5878a54afb2f51a0f22deb3b"} Sep 17 20:58:49.581440 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.581344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerStarted","Data":"d673f775ba129052d5fe236b48cf63d1745fa15486bde65c6bb07ef2d5d3a224"} Sep 17 20:58:49.610259 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:49.610176 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d"] Sep 17 20:58:49.612535 ip-10-0-140-118 kubenswrapper[2579]: W0917 20:58:49.612512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf379d2_56bc_4856_96cc_3bfba4ea51fc.slice/crio-17c237cd496caed00902ca81e7fc5970791ad2cd4f7a2db0f429dcd3036aa5aa WatchSource:0}: Error finding container 17c237cd496caed00902ca81e7fc5970791ad2cd4f7a2db0f429dcd3036aa5aa: Status 404 returned error can't find the container with id 17c237cd496caed00902ca81e7fc5970791ad2cd4f7a2db0f429dcd3036aa5aa Sep 17 20:58:50.587745 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:50.587710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerStarted","Data":"56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58"} Sep 17 20:58:50.589518 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:50.589485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerStarted","Data":"01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290"} Sep 17 20:58:50.589620 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:50.589560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerStarted","Data":"17c237cd496caed00902ca81e7fc5970791ad2cd4f7a2db0f429dcd3036aa5aa"} Sep 17 20:58:53.607082 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:53.607044 2579 generic.go:356] "Generic (PLEG): container finished" podID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerID="01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290" exitCode=0 Sep 17 20:58:53.607501 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:53.607118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerDied","Data":"01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290"} Sep 17 20:58:53.608941 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:53.608908 2579 generic.go:356] "Generic (PLEG): container finished" podID="b726354c-87d0-4a8c-9768-1c66f236f316" containerID="9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a" exitCode=0 Sep 17 20:58:53.609057 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:53.608946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerDied","Data":"9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a"} Sep 17 20:58:53.610721 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:53.610659 2579 generic.go:356] "Generic (PLEG): container finished" podID="7534274d-cffc-4e7c-b216-b7141ee17518" containerID="56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58" exitCode=0 Sep 17 20:58:53.610721 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:58:53.610701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerDied","Data":"56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58"} Sep 17 20:59:23.845854 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:23.845813 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerStarted","Data":"fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166"} Sep 17 20:59:23.849651 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:23.849620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerStarted","Data":"45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0"} Sep 17 20:59:23.853759 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:23.853728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerStarted","Data":"31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04"} Sep 17 20:59:26.872383 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.872329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerStarted","Data":"e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780"} Sep 17 20:59:26.872887 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.872429 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:59:26.872887 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.872464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:59:26.874160 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.874126 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 20:59:26.874661 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.874641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerStarted","Data":"bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed"} Sep 17 20:59:26.874851 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.874831 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:59:26.874925 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.874882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:59:26.876019 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.875978 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 20:59:26.877105 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.877082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerStarted","Data":"ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77"} Sep 17 20:59:26.877217 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.877202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:59:26.877280 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.877240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:59:26.878128 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.878104 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 20:59:26.895293 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.895232 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podStartSLOduration=2.90101356 podStartE2EDuration="38.895212849s" totalImagesPullingTime="32.380256908s" totalInitContainerRuntime="3.613942381s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:26.892190868 +0000 UTC m=+612.366873214" watchObservedRunningTime="2026-09-17 20:59:26.895212849 +0000 UTC m=+612.369895194" Sep 17 20:59:26.912939 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.912897 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podStartSLOduration=2.786157805 podStartE2EDuration="38.912883493s" totalImagesPullingTime="32.509980923s" totalInitContainerRuntime="3.616744765s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:26.911572292 +0000 UTC m=+612.386254637" watchObservedRunningTime="2026-09-17 20:59:26.912883493 +0000 UTC m=+612.387565887" Sep 17 20:59:26.931466 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:26.931421 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podStartSLOduration=2.718219887 podStartE2EDuration="38.931408443s" totalImagesPullingTime="32.430969913s" totalInitContainerRuntime="3.782218643s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:26.929705229 +0000 UTC m=+612.404387583" watchObservedRunningTime="2026-09-17 20:59:26.931408443 +0000 UTC m=+612.406090786" Sep 17 20:59:27.882471 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:27.882412 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 20:59:27.882950 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:27.882546 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 20:59:27.882950 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:27.882579 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 20:59:32.887308 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:32.887275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 20:59:32.887873 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:32.887804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 20:59:32.887873 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:32.887829 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 20:59:32.887954 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:32.887882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 20:59:32.888270 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:32.888244 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 20:59:32.888314 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:32.888244 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 20:59:42.888034 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:42.887948 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 20:59:42.888432 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:42.888182 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 20:59:42.888432 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:42.888201 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 20:59:52.887765 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:52.887704 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 20:59:52.888220 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:52.888165 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 20:59:52.888303 ip-10-0-140-118 kubenswrapper[2579]: I0917 20:59:52.888280 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 21:00:02.888406 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:02.888353 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 21:00:02.888406 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:02.888395 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 21:00:02.888847 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:02.888396 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 21:00:12.887726 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:12.887692 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 21:00:12.888957 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:12.888932 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 21:00:12.889033 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:12.888934 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 21:00:22.888207 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:22.888164 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 21:00:22.888637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:22.888341 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 21:00:22.888637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:22.888342 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 21:00:32.888235 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:32.888195 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 21:00:32.888235 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:32.888210 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 21:00:32.888857 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:32.888838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 21:00:36.048461 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:36.048421 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 17 21:00:42.888533 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:42.888506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 21:00:46.048541 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:00:46.048507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 21:01:19.846942 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.846863 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d"] Sep 17 21:01:19.847550 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.847128 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" containerID="cri-o://31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04" gracePeriod=30 Sep 17 21:01:19.847550 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.847157 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kube-rbac-proxy" containerID="cri-o://e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780" gracePeriod=30 Sep 17 21:01:19.887254 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.887228 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps"] Sep 17 21:01:19.887536 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.887495 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" containerID="cri-o://45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0" gracePeriod=30 Sep 17 21:01:19.887643 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.887568 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kube-rbac-proxy" containerID="cri-o://ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77" gracePeriod=30 Sep 17 21:01:19.920642 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.920618 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5"] Sep 17 21:01:19.920874 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.920837 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" containerID="cri-o://fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166" gracePeriod=30 Sep 17 21:01:19.920934 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:19.920887 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kube-rbac-proxy" containerID="cri-o://bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed" gracePeriod=30 Sep 17 21:01:20.378795 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:20.378762 2579 generic.go:356] "Generic (PLEG): container finished" podID="b726354c-87d0-4a8c-9768-1c66f236f316" containerID="bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed" exitCode=2 Sep 17 21:01:20.378948 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:20.378829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerDied","Data":"bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed"} Sep 17 21:01:20.380934 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:20.380912 2579 generic.go:356] "Generic (PLEG): container finished" podID="7534274d-cffc-4e7c-b216-b7141ee17518" containerID="ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77" exitCode=2 Sep 17 21:01:20.381069 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:20.380996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerDied","Data":"ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77"} Sep 17 21:01:20.382964 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:20.382946 2579 generic.go:356] "Generic (PLEG): container finished" podID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerID="e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780" exitCode=2 Sep 17 21:01:20.383036 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:20.382995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerDied","Data":"e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780"} Sep 17 21:01:22.883493 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:22.883449 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 17 21:01:22.883862 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:22.883450 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 21:01:22.883862 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:22.883450 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused" Sep 17 21:01:22.888189 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:22.888162 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 21:01:22.888327 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:22.888259 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 21:01:23.863080 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.863059 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 21:01:23.950598 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.950536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b726354c-87d0-4a8c-9768-1c66f236f316-proxy-tls\") pod \"b726354c-87d0-4a8c-9768-1c66f236f316\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " Sep 17 21:01:23.950598 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.950574 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-cabundle-cert\") pod \"b726354c-87d0-4a8c-9768-1c66f236f316\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " Sep 17 21:01:23.951053 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.950627 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttq84\" (UniqueName: \"kubernetes.io/projected/b726354c-87d0-4a8c-9768-1c66f236f316-kube-api-access-ttq84\") pod \"b726354c-87d0-4a8c-9768-1c66f236f316\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " Sep 17 21:01:23.951053 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.950663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b726354c-87d0-4a8c-9768-1c66f236f316-kserve-provision-location\") pod \"b726354c-87d0-4a8c-9768-1c66f236f316\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " Sep 17 21:01:23.951053 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.950692 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-kube-rbac-proxy-sar-config\") pod \"b726354c-87d0-4a8c-9768-1c66f236f316\" (UID: \"b726354c-87d0-4a8c-9768-1c66f236f316\") " Sep 17 21:01:23.951209 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.951055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b726354c-87d0-4a8c-9768-1c66f236f316-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b726354c-87d0-4a8c-9768-1c66f236f316" (UID: "b726354c-87d0-4a8c-9768-1c66f236f316"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 21:01:23.951209 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.951104 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b726354c-87d0-4a8c-9768-1c66f236f316" (UID: "b726354c-87d0-4a8c-9768-1c66f236f316"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 21:01:23.951209 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.951141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b726354c-87d0-4a8c-9768-1c66f236f316" (UID: "b726354c-87d0-4a8c-9768-1c66f236f316"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 21:01:23.952900 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.952882 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b726354c-87d0-4a8c-9768-1c66f236f316-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b726354c-87d0-4a8c-9768-1c66f236f316" (UID: "b726354c-87d0-4a8c-9768-1c66f236f316"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 21:01:23.953090 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:23.953076 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b726354c-87d0-4a8c-9768-1c66f236f316-kube-api-access-ttq84" (OuterVolumeSpecName: "kube-api-access-ttq84") pod "b726354c-87d0-4a8c-9768-1c66f236f316" (UID: "b726354c-87d0-4a8c-9768-1c66f236f316"). InnerVolumeSpecName "kube-api-access-ttq84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 21:01:24.051588 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.051567 2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b726354c-87d0-4a8c-9768-1c66f236f316-kserve-provision-location\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.051588 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.051586 2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.051723 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.051596 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b726354c-87d0-4a8c-9768-1c66f236f316-proxy-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.051723 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.051605 2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b726354c-87d0-4a8c-9768-1c66f236f316-cabundle-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.051723 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.051613 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttq84\" (UniqueName: \"kubernetes.io/projected/b726354c-87d0-4a8c-9768-1c66f236f316-kube-api-access-ttq84\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.405085 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.405050 2579 generic.go:356] "Generic (PLEG): container finished" podID="b726354c-87d0-4a8c-9768-1c66f236f316" containerID="fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166" exitCode=0 Sep 17 21:01:24.405271 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.405109 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" Sep 17 21:01:24.405271 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.405138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerDied","Data":"fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166"} Sep 17 21:01:24.405271 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.405209 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5" event={"ID":"b726354c-87d0-4a8c-9768-1c66f236f316","Type":"ContainerDied","Data":"fe1d29001c4ea63287d59b5898b447f8969249ec5878a54afb2f51a0f22deb3b"} Sep 17 21:01:24.405271 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.405247 2579 scope.go:117] "RemoveContainer" containerID="bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed" Sep 17 21:01:24.414572 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.414559 2579 scope.go:117] "RemoveContainer" containerID="fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166" Sep 17 21:01:24.421852 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.421836 2579 scope.go:117] "RemoveContainer" containerID="9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a" Sep 17 21:01:24.427896 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.427874 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5"] Sep 17 21:01:24.429038 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.429023 2579 scope.go:117] "RemoveContainer" containerID="bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed" Sep 17 21:01:24.429334 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:24.429316 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed\": container with ID starting with bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed not found: ID does not exist" containerID="bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed" Sep 17 21:01:24.429401 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.429343 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed"} err="failed to get container status \"bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed\": rpc error: code = NotFound desc = could not find container \"bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed\": container with ID starting with bf8927b4f5f5887979f20391d5e711274843793222f9fbf7335f116b92f631ed not found: ID does not exist" Sep 17 21:01:24.429401 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.429359 2579 scope.go:117] "RemoveContainer" containerID="fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166" Sep 17 21:01:24.429675 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:24.429657 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166\": container with ID starting with fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166 not found: ID does not exist" containerID="fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166" Sep 17 21:01:24.429738 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.429683 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166"} err="failed to get container status \"fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166\": rpc error: code = NotFound desc = could not find container \"fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166\": container with ID starting with fc1b867deb2db0e8e066b011c4161078516b9d9b3acd8f521503f3df25d43166 not found: ID does not exist" Sep 17 21:01:24.429738 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.429707 2579 scope.go:117] "RemoveContainer" containerID="9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a" Sep 17 21:01:24.429976 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:24.429959 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a\": container with ID starting with 9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a not found: ID does not exist" containerID="9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a" Sep 17 21:01:24.430016 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.429982 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a"} err="failed to get container status \"9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a\": rpc error: code = NotFound desc = could not find container \"9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a\": container with ID starting with 9a0984a61d7c74e850ef3bd4af140327893b2888cbc6198a858c146c5cc4f73a not found: ID does not exist" Sep 17 21:01:24.435821 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.435803 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5"] Sep 17 21:01:24.855207 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.855181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 21:01:24.857740 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.857720 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 21:01:24.959905 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.959852 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7534274d-cffc-4e7c-b216-b7141ee17518-kserve-provision-location\") pod \"7534274d-cffc-4e7c-b216-b7141ee17518\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " Sep 17 21:01:24.959905 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.959886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kserve-provision-location\") pod \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.959909 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-cabundle-cert\") pod \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.959935 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls\") pod \"7534274d-cffc-4e7c-b216-b7141ee17518\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.959953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-rbac-proxy-sar-config\") pod \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.959982 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-cabundle-cert\") pod \"7534274d-cffc-4e7c-b216-b7141ee17518\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960190 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66r2f\" (UniqueName: \"kubernetes.io/projected/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-api-access-66r2f\") pod \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960202 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7534274d-cffc-4e7c-b216-b7141ee17518-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7534274d-cffc-4e7c-b216-b7141ee17518" (UID: "7534274d-cffc-4e7c-b216-b7141ee17518"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960278 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daf379d2-56bc-4856-96cc-3bfba4ea51fc" (UID: "daf379d2-56bc-4856-96cc-3bfba4ea51fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960250 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-kube-rbac-proxy-sar-config\") pod \"7534274d-cffc-4e7c-b216-b7141ee17518\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960329 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "daf379d2-56bc-4856-96cc-3bfba4ea51fc" (UID: "daf379d2-56bc-4856-96cc-3bfba4ea51fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960334 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7534274d-cffc-4e7c-b216-b7141ee17518" (UID: "7534274d-cffc-4e7c-b216-b7141ee17518"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxrp6\" (UniqueName: \"kubernetes.io/projected/7534274d-cffc-4e7c-b216-b7141ee17518-kube-api-access-cxrp6\") pod \"7534274d-cffc-4e7c-b216-b7141ee17518\" (UID: \"7534274d-cffc-4e7c-b216-b7141ee17518\") " Sep 17 21:01:24.960472 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf379d2-56bc-4856-96cc-3bfba4ea51fc-proxy-tls\") pod \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\" (UID: \"daf379d2-56bc-4856-96cc-3bfba4ea51fc\") " Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960500 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daf379d2-56bc-4856-96cc-3bfba4ea51fc" (UID: "daf379d2-56bc-4856-96cc-3bfba4ea51fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7534274d-cffc-4e7c-b216-b7141ee17518" (UID: "7534274d-cffc-4e7c-b216-b7141ee17518"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960680 2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7534274d-cffc-4e7c-b216-b7141ee17518-kserve-provision-location\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960697 2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kserve-provision-location\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960713 2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-cabundle-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960726 2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960739 2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-cabundle-cert\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.961637 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.960752 2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7534274d-cffc-4e7c-b216-b7141ee17518-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:24.962541 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.962517 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-api-access-66r2f" (OuterVolumeSpecName: "kube-api-access-66r2f") pod "daf379d2-56bc-4856-96cc-3bfba4ea51fc" (UID: "daf379d2-56bc-4856-96cc-3bfba4ea51fc"). InnerVolumeSpecName "kube-api-access-66r2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 21:01:24.962628 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.962543 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf379d2-56bc-4856-96cc-3bfba4ea51fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daf379d2-56bc-4856-96cc-3bfba4ea51fc" (UID: "daf379d2-56bc-4856-96cc-3bfba4ea51fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 21:01:24.962628 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.962566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7534274d-cffc-4e7c-b216-b7141ee17518-kube-api-access-cxrp6" (OuterVolumeSpecName: "kube-api-access-cxrp6") pod "7534274d-cffc-4e7c-b216-b7141ee17518" (UID: "7534274d-cffc-4e7c-b216-b7141ee17518"). InnerVolumeSpecName "kube-api-access-cxrp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 21:01:24.962628 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:24.962609 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7534274d-cffc-4e7c-b216-b7141ee17518" (UID: "7534274d-cffc-4e7c-b216-b7141ee17518"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 21:01:25.053477 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.053450 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" path="/var/lib/kubelet/pods/b726354c-87d0-4a8c-9768-1c66f236f316/volumes" Sep 17 21:01:25.061208 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.061192 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66r2f\" (UniqueName: \"kubernetes.io/projected/daf379d2-56bc-4856-96cc-3bfba4ea51fc-kube-api-access-66r2f\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:25.061269 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.061211 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrp6\" (UniqueName: \"kubernetes.io/projected/7534274d-cffc-4e7c-b216-b7141ee17518-kube-api-access-cxrp6\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:25.061269 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.061220 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf379d2-56bc-4856-96cc-3bfba4ea51fc-proxy-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:25.061269 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.061229 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7534274d-cffc-4e7c-b216-b7141ee17518-proxy-tls\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:01:25.411355 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.411323 2579 generic.go:356] "Generic (PLEG): container finished" podID="7534274d-cffc-4e7c-b216-b7141ee17518" containerID="45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0" exitCode=0 Sep 17 21:01:25.411488 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.411442 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" Sep 17 21:01:25.411488 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.411469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerDied","Data":"45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0"} Sep 17 21:01:25.411614 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.411542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps" event={"ID":"7534274d-cffc-4e7c-b216-b7141ee17518","Type":"ContainerDied","Data":"d673f775ba129052d5fe236b48cf63d1745fa15486bde65c6bb07ef2d5d3a224"} Sep 17 21:01:25.411614 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.411574 2579 scope.go:117] "RemoveContainer" containerID="ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77" Sep 17 21:01:25.415157 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.415135 2579 generic.go:356] "Generic (PLEG): container finished" podID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerID="31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04" exitCode=0 Sep 17 21:01:25.415421 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.415402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerDied","Data":"31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04"} Sep 17 21:01:25.415628 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.415608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" event={"ID":"daf379d2-56bc-4856-96cc-3bfba4ea51fc","Type":"ContainerDied","Data":"17c237cd496caed00902ca81e7fc5970791ad2cd4f7a2db0f429dcd3036aa5aa"} Sep 17 21:01:25.415794 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.415776 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d" Sep 17 21:01:25.422916 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.422898 2579 scope.go:117] "RemoveContainer" containerID="45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0" Sep 17 21:01:25.430036 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.430016 2579 scope.go:117] "RemoveContainer" containerID="56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58" Sep 17 21:01:25.431923 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.431900 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps"] Sep 17 21:01:25.435998 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.435979 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps"] Sep 17 21:01:25.437703 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.437686 2579 scope.go:117] "RemoveContainer" containerID="ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77" Sep 17 21:01:25.437952 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:25.437936 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77\": container with ID starting with ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77 not found: ID does not exist" containerID="ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77" Sep 17 21:01:25.438023 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.437965 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77"} err="failed to get container status \"ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77\": rpc error: code = NotFound desc = could not find container \"ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77\": container with ID starting with ca16650f133c405ebc49d125c9f077715cb1549ac13cdd1bc546e1eff4afea77 not found: ID does not exist" Sep 17 21:01:25.438023 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.437987 2579 scope.go:117] "RemoveContainer" containerID="45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0" Sep 17 21:01:25.438244 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:25.438228 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0\": container with ID starting with 45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0 not found: ID does not exist" containerID="45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0" Sep 17 21:01:25.438244 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.438251 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0"} err="failed to get container status \"45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0\": rpc error: code = NotFound desc = could not find container \"45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0\": container with ID starting with 45118641c322578f34d235945899ed5bf92226ffa40c92e07bdd4cf5e49ddab0 not found: ID does not exist" Sep 17 21:01:25.438244 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.438266 2579 scope.go:117] "RemoveContainer" containerID="56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58" Sep 17 21:01:25.438530 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:25.438516 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58\": container with ID starting with 56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58 not found: ID does not exist" containerID="56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58" Sep 17 21:01:25.438573 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.438533 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58"} err="failed to get container status \"56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58\": rpc error: code = NotFound desc = could not find container \"56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58\": container with ID starting with 56be0c72fca6ac65a433f47dac7323c49ca69f5c23d713a4ff18fca2f9199d58 not found: ID does not exist" Sep 17 21:01:25.438573 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.438546 2579 scope.go:117] "RemoveContainer" containerID="e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780" Sep 17 21:01:25.445558 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.445506 2579 scope.go:117] "RemoveContainer" containerID="31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04" Sep 17 21:01:25.447332 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.447317 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d"] Sep 17 21:01:25.452062 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.452041 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d"] Sep 17 21:01:25.453143 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.453126 2579 scope.go:117] "RemoveContainer" containerID="01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290" Sep 17 21:01:25.459989 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.459974 2579 scope.go:117] "RemoveContainer" containerID="e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780" Sep 17 21:01:25.460237 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:25.460220 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780\": container with ID starting with e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780 not found: ID does not exist" containerID="e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780" Sep 17 21:01:25.460294 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.460244 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780"} err="failed to get container status \"e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780\": rpc error: code = NotFound desc = could not find container \"e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780\": container with ID starting with e0f8db17597cf2bb1c33032efff7ed8a1fac9137c0c6b086a421578c40e93780 not found: ID does not exist" Sep 17 21:01:25.460294 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.460258 2579 scope.go:117] "RemoveContainer" containerID="31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04" Sep 17 21:01:25.460559 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:25.460542 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04\": container with ID starting with 31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04 not found: ID does not exist" containerID="31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04" Sep 17 21:01:25.460612 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.460565 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04"} err="failed to get container status \"31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04\": rpc error: code = NotFound desc = could not find container \"31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04\": container with ID starting with 31fbea5504de616a753cfa973a934199071bd07c4bc91c7532d3b79e5adc2b04 not found: ID does not exist" Sep 17 21:01:25.460612 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.460580 2579 scope.go:117] "RemoveContainer" containerID="01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290" Sep 17 21:01:25.460816 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:01:25.460798 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290\": container with ID starting with 01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290 not found: ID does not exist" containerID="01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290" Sep 17 21:01:25.460897 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:25.460818 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290"} err="failed to get container status \"01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290\": rpc error: code = NotFound desc = could not find container \"01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290\": container with ID starting with 01bcba578e61acb072bec7c76d46ff3017ef01f707aed6cf21dbfeb07a274290 not found: ID does not exist" Sep 17 21:01:27.053308 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:27.053264 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" path="/var/lib/kubelet/pods/7534274d-cffc-4e7c-b216-b7141ee17518/volumes" Sep 17 21:01:27.053873 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:01:27.053854 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" path="/var/lib/kubelet/pods/daf379d2-56bc-4856-96cc-3bfba4ea51fc/volumes" Sep 17 21:39:03.904778 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.904739 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4mcpc/must-gather-fvfzx"] Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905191 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905207 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905218 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="storage-initializer" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905224 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="storage-initializer" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905230 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905235 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905247 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905254 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905264 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905269 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905278 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905283 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905290 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="storage-initializer" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905295 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="storage-initializer" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905301 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905306 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905313 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="storage-initializer" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905317 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="storage-initializer" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905386 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905396 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905403 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kube-rbac-proxy" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905410 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7534274d-cffc-4e7c-b216-b7141ee17518" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905416 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b726354c-87d0-4a8c-9768-1c66f236f316" containerName="kserve-container" Sep 17 21:39:03.907359 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.905422 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf379d2-56bc-4856-96cc-3bfba4ea51fc" containerName="kserve-container" Sep 17 21:39:03.908303 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.908286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:03.913276 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.913253 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4mcpc\"/\"openshift-service-ca.crt\"" Sep 17 21:39:03.913421 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.913328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4mcpc\"/\"kube-root-ca.crt\"" Sep 17 21:39:03.914952 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.914933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4mcpc\"/\"default-dockercfg-dttpb\"" Sep 17 21:39:03.923984 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:03.923964 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4mcpc/must-gather-fvfzx"] Sep 17 21:39:04.002121 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.002103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/977d629a-1677-4bd5-9d95-b110823a83c1-must-gather-output\") pod \"must-gather-fvfzx\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.002203 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.002133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5j2z\" (UniqueName: \"kubernetes.io/projected/977d629a-1677-4bd5-9d95-b110823a83c1-kube-api-access-w5j2z\") pod \"must-gather-fvfzx\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.103420 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.103389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/977d629a-1677-4bd5-9d95-b110823a83c1-must-gather-output\") pod \"must-gather-fvfzx\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.103506 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.103431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5j2z\" (UniqueName: \"kubernetes.io/projected/977d629a-1677-4bd5-9d95-b110823a83c1-kube-api-access-w5j2z\") pod \"must-gather-fvfzx\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.103701 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.103683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/977d629a-1677-4bd5-9d95-b110823a83c1-must-gather-output\") pod \"must-gather-fvfzx\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.112403 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.112358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5j2z\" (UniqueName: \"kubernetes.io/projected/977d629a-1677-4bd5-9d95-b110823a83c1-kube-api-access-w5j2z\") pod \"must-gather-fvfzx\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.216613 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.216559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:04.497267 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.497243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4mcpc/must-gather-fvfzx"] Sep 17 21:39:04.499805 ip-10-0-140-118 kubenswrapper[2579]: W0917 21:39:04.499779 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977d629a_1677_4bd5_9d95_b110823a83c1.slice/crio-85661c533b3f7ce030b285f8c47ff7b64e7efab1ea4d1b581052dc49f4cdc189 WatchSource:0}: Error finding container 85661c533b3f7ce030b285f8c47ff7b64e7efab1ea4d1b581052dc49f4cdc189: Status 404 returned error can't find the container with id 85661c533b3f7ce030b285f8c47ff7b64e7efab1ea4d1b581052dc49f4cdc189 Sep 17 21:39:04.501729 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:04.501709 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 21:39:05.484523 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:05.484481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" event={"ID":"977d629a-1677-4bd5-9d95-b110823a83c1","Type":"ContainerStarted","Data":"85661c533b3f7ce030b285f8c47ff7b64e7efab1ea4d1b581052dc49f4cdc189"} Sep 17 21:39:11.513091 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:11.513056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" event={"ID":"977d629a-1677-4bd5-9d95-b110823a83c1","Type":"ContainerStarted","Data":"efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c"} Sep 17 21:39:11.513091 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:11.513101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" event={"ID":"977d629a-1677-4bd5-9d95-b110823a83c1","Type":"ContainerStarted","Data":"21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb"} Sep 17 21:39:11.530813 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:11.530768 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" podStartSLOduration=2.426977577 podStartE2EDuration="8.530753921s" totalImagesPullingTime="6.103776344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:39:11.52910204 +0000 UTC m=+2997.003784384" watchObservedRunningTime="2026-09-17 21:39:11.530753921 +0000 UTC m=+2997.005436265" Sep 17 21:39:34.629689 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:34.629654 2579 generic.go:356] "Generic (PLEG): container finished" podID="977d629a-1677-4bd5-9d95-b110823a83c1" containerID="21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb" exitCode=0 Sep 17 21:39:34.630127 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:34.629732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" event={"ID":"977d629a-1677-4bd5-9d95-b110823a83c1","Type":"ContainerDied","Data":"21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb"} Sep 17 21:39:34.630127 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:34.630055 2579 scope.go:117] "RemoveContainer" containerID="21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb" Sep 17 21:39:34.863480 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:34.863459 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4mcpc_must-gather-fvfzx_977d629a-1677-4bd5-9d95-b110823a83c1/gather/0.log" Sep 17 21:39:38.314336 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:38.314305 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xqdcz_9bca1fc5-af86-4106-b5f5-b7ffdefab53a/global-pull-secret-syncer/0.log" Sep 17 21:39:38.395258 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:38.395236 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gb8f6_d769f3fe-96c8-4c71-9890-6d16a4869ec1/konnectivity-agent/0.log" Sep 17 21:39:38.503842 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:38.503817 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-118.ec2.internal_576eb62884aeb095e963216a64f67066/haproxy/0.log" Sep 17 21:39:40.288750 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.288707 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-4mcpc/must-gather-fvfzx"] Sep 17 21:39:40.289334 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.288916 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="copy" containerID="cri-o://efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c" gracePeriod=2 Sep 17 21:39:40.292127 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:39:40.292087 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fvfzx\" is forbidden: User \"system:node:ip-10-0-140-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4mcpc\": no relationship found between node 'ip-10-0-140-118.ec2.internal' and this object" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:40.294327 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.293652 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-4mcpc/must-gather-fvfzx"] Sep 17 21:39:40.528623 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.528599 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4mcpc_must-gather-fvfzx_977d629a-1677-4bd5-9d95-b110823a83c1/copy/0.log" Sep 17 21:39:40.531178 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.531161 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:40.533712 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:39:40.533688 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fvfzx\" is forbidden: User \"system:node:ip-10-0-140-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4mcpc\": no relationship found between node 'ip-10-0-140-118.ec2.internal' and this object" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:40.620841 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.620791 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5j2z\" (UniqueName: \"kubernetes.io/projected/977d629a-1677-4bd5-9d95-b110823a83c1-kube-api-access-w5j2z\") pod \"977d629a-1677-4bd5-9d95-b110823a83c1\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " Sep 17 21:39:40.620933 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.620843 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/977d629a-1677-4bd5-9d95-b110823a83c1-must-gather-output\") pod \"977d629a-1677-4bd5-9d95-b110823a83c1\" (UID: \"977d629a-1677-4bd5-9d95-b110823a83c1\") " Sep 17 21:39:40.622409 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.622385 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977d629a-1677-4bd5-9d95-b110823a83c1-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "977d629a-1677-4bd5-9d95-b110823a83c1" (UID: "977d629a-1677-4bd5-9d95-b110823a83c1"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 21:39:40.623006 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.622987 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977d629a-1677-4bd5-9d95-b110823a83c1-kube-api-access-w5j2z" (OuterVolumeSpecName: "kube-api-access-w5j2z") pod "977d629a-1677-4bd5-9d95-b110823a83c1" (UID: "977d629a-1677-4bd5-9d95-b110823a83c1"). InnerVolumeSpecName "kube-api-access-w5j2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 21:39:40.657590 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.657572 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4mcpc_must-gather-fvfzx_977d629a-1677-4bd5-9d95-b110823a83c1/copy/0.log" Sep 17 21:39:40.657931 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.657911 2579 generic.go:356] "Generic (PLEG): container finished" podID="977d629a-1677-4bd5-9d95-b110823a83c1" containerID="efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c" exitCode=143 Sep 17 21:39:40.657999 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.657955 2579 scope.go:117] "RemoveContainer" containerID="efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c" Sep 17 21:39:40.658058 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.657958 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:40.660766 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:39:40.660739 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fvfzx\" is forbidden: User \"system:node:ip-10-0-140-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4mcpc\": no relationship found between node 'ip-10-0-140-118.ec2.internal' and this object" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:40.665844 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.665824 2579 scope.go:117] "RemoveContainer" containerID="21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb" Sep 17 21:39:40.668544 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:39:40.668517 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fvfzx\" is forbidden: User \"system:node:ip-10-0-140-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4mcpc\": no relationship found between node 'ip-10-0-140-118.ec2.internal' and this object" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" pod="openshift-must-gather-4mcpc/must-gather-fvfzx" Sep 17 21:39:40.677812 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.677794 2579 scope.go:117] "RemoveContainer" containerID="efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c" Sep 17 21:39:40.678081 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:39:40.678063 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c\": container with ID starting with efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c not found: ID does not exist" containerID="efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c" Sep 17 21:39:40.678123 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.678089 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c"} err="failed to get container status \"efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c\": rpc error: code = NotFound desc = could not find container \"efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c\": container with ID starting with efb903058390e4be3a9fc6b05a28da679e07e4dfc4839efec5dc920556cf171c not found: ID does not exist" Sep 17 21:39:40.678123 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.678106 2579 scope.go:117] "RemoveContainer" containerID="21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb" Sep 17 21:39:40.678343 ip-10-0-140-118 kubenswrapper[2579]: E0917 21:39:40.678325 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb\": container with ID starting with 21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb not found: ID does not exist" containerID="21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb" Sep 17 21:39:40.678404 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.678350 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb"} err="failed to get container status \"21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb\": rpc error: code = NotFound desc = could not find container \"21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb\": container with ID starting with 21712647ffb6aeddacec2ffe88cf28f5f72e7b6801a06e81fdb22101394398fb not found: ID does not exist" Sep 17 21:39:40.721910 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.721887 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5j2z\" (UniqueName: \"kubernetes.io/projected/977d629a-1677-4bd5-9d95-b110823a83c1-kube-api-access-w5j2z\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:39:40.721910 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:40.721908 2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/977d629a-1677-4bd5-9d95-b110823a83c1-must-gather-output\") on node \"ip-10-0-140-118.ec2.internal\" DevicePath \"\"" Sep 17 21:39:41.053843 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:41.053811 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" path="/var/lib/kubelet/pods/977d629a-1677-4bd5-9d95-b110823a83c1/volumes" Sep 17 21:39:42.878752 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:42.878729 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/alertmanager/0.log" Sep 17 21:39:42.925819 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:42.925801 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/config-reloader/0.log" Sep 17 21:39:42.957049 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:42.957024 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/kube-rbac-proxy-web/0.log" Sep 17 21:39:43.000410 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.000390 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/kube-rbac-proxy/0.log" Sep 17 21:39:43.032800 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.032783 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/kube-rbac-proxy-metric/0.log" Sep 17 21:39:43.066248 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.066233 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/prom-label-proxy/0.log" Sep 17 21:39:43.092090 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.092071 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ea96a6e2-af38-4510-8d3d-fc2353f9a78d/init-config-reloader/0.log" Sep 17 21:39:43.135434 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.135383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-h9drm_2f0c9995-a015-4054-8ba6-0b39ae783f5e/cluster-monitoring-operator/0.log" Sep 17 21:39:43.386636 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.386577 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-rxx9b_ad16d675-8e2d-4f9c-be85-4933217f2f81/monitoring-plugin/0.log" Sep 17 21:39:43.489763 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.489742 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qccf_1c3157ff-c8e0-4144-9aee-7e12931611f7/node-exporter/0.log" Sep 17 21:39:43.539702 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.539687 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qccf_1c3157ff-c8e0-4144-9aee-7e12931611f7/kube-rbac-proxy/0.log" Sep 17 21:39:43.572799 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.572782 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qccf_1c3157ff-c8e0-4144-9aee-7e12931611f7/init-textfile/0.log" Sep 17 21:39:43.836256 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.836224 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-cx6dd_fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e/kube-rbac-proxy-main/0.log" Sep 17 21:39:43.871665 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.871642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-cx6dd_fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e/kube-rbac-proxy-self/0.log" Sep 17 21:39:43.920467 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.920448 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-cx6dd_fe1a9dfc-e2f3-4746-98f9-dc6439bc3b4e/openshift-state-metrics/0.log" Sep 17 21:39:43.976350 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:43.976329 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/prometheus/0.log" Sep 17 21:39:44.010940 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.010918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/config-reloader/0.log" Sep 17 21:39:44.038273 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.038257 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/thanos-sidecar/0.log" Sep 17 21:39:44.066917 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.066903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/kube-rbac-proxy-web/0.log" Sep 17 21:39:44.131898 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.131849 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/kube-rbac-proxy/0.log" Sep 17 21:39:44.183740 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.183722 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/kube-rbac-proxy-thanos/0.log" Sep 17 21:39:44.233641 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.233623 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fff22ab4-3ac9-462e-92aa-83931293e56a/init-config-reloader/0.log" Sep 17 21:39:44.463727 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:44.463696 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-g4hhp_714314bb-6de6-47e0-ab08-d3ee0f4a2de1/prometheus-operator-admission-webhook/0.log" Sep 17 21:39:45.779287 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.779248 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-6fhw8_d643d2e6-72f0-470f-b84d-d225769d6bd5/networking-console-plugin/0.log" Sep 17 21:39:45.811663 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.811636 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp"] Sep 17 21:39:45.811935 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.811923 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="gather" Sep 17 21:39:45.811986 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.811937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="gather" Sep 17 21:39:45.811986 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.811948 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="copy" Sep 17 21:39:45.811986 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.811953 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="copy" Sep 17 21:39:45.812081 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.812014 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="gather" Sep 17 21:39:45.812081 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.812023 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d629a-1677-4bd5-9d95-b110823a83c1" containerName="copy" Sep 17 21:39:45.817132 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.817113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.820191 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.820172 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-44q44\"/\"kube-root-ca.crt\"" Sep 17 21:39:45.821653 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.821636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-44q44\"/\"openshift-service-ca.crt\"" Sep 17 21:39:45.821717 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.821636 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-44q44\"/\"default-dockercfg-rvlpj\"" Sep 17 21:39:45.824208 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.823799 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp"] Sep 17 21:39:45.859269 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.859249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-podres\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.859406 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.859294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-lib-modules\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.859406 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.859312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-sys\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.859406 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.859330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-proc\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.859529 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.859440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5tk\" (UniqueName: \"kubernetes.io/projected/05b9fdaf-50e6-454a-8126-25087905f2f5-kube-api-access-ww5tk\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960297 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5tk\" (UniqueName: \"kubernetes.io/projected/05b9fdaf-50e6-454a-8126-25087905f2f5-kube-api-access-ww5tk\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960454 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-podres\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960454 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-lib-modules\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960454 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-sys\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960454 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-proc\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960667 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-podres\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960667 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-lib-modules\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960667 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-sys\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.960667 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.960527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05b9fdaf-50e6-454a-8126-25087905f2f5-proc\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:45.969764 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:45.969740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5tk\" (UniqueName: \"kubernetes.io/projected/05b9fdaf-50e6-454a-8126-25087905f2f5-kube-api-access-ww5tk\") pod \"perf-node-gather-daemonset-h9ttp\" (UID: \"05b9fdaf-50e6-454a-8126-25087905f2f5\") " pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:46.127852 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:46.127786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:46.408186 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:46.408161 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp"] Sep 17 21:39:46.681895 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:46.681834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" event={"ID":"05b9fdaf-50e6-454a-8126-25087905f2f5","Type":"ContainerStarted","Data":"5c459cdfcdbab3c27c35533b2ca8e2f26bae491782a0a521a9234b709cbae419"} Sep 17 21:39:46.681895 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:46.681881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" event={"ID":"05b9fdaf-50e6-454a-8126-25087905f2f5","Type":"ContainerStarted","Data":"ab77d4e983be893d4e69d26d5adfcc105b733945696ed4476814ca9838b640bd"} Sep 17 21:39:46.682028 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:46.682010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:46.700929 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:46.700890 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" podStartSLOduration=1.700877131 podStartE2EDuration="1.700877131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:39:46.699552513 +0000 UTC m=+3032.174234857" watchObservedRunningTime="2026-09-17 21:39:46.700877131 +0000 UTC m=+3032.175559475" Sep 17 21:39:48.170878 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:48.170847 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8jqfh_74fbf86b-d686-4752-8c33-1607b699dece/dns/0.log" Sep 17 21:39:48.217006 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:48.216981 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8jqfh_74fbf86b-d686-4752-8c33-1607b699dece/kube-rbac-proxy/0.log" Sep 17 21:39:48.425871 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:48.425813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgqm7_fc6dd067-c1f7-44ad-9ec7-62bf9738c142/dns-node-resolver/0.log" Sep 17 21:39:48.907339 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:48.907315 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8vvzl_387082f5-712b-4e1c-820e-66c7f71a0250/node-ca/0.log" Sep 17 21:39:50.095904 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:50.095872 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nt4pt_d7d3912c-f311-4cda-b935-cf83db10c7c4/serve-healthcheck-canary/0.log" Sep 17 21:39:50.460258 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:50.460181 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-shcng_8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9/insights-operator/0.log" Sep 17 21:39:50.460607 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:50.460590 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-shcng_8bbe1c9e-0f9e-44f2-b3d0-530393ef4ed9/insights-operator/1.log" Sep 17 21:39:50.554912 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:50.554888 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsw4z_1b6918ad-3c4a-445e-932a-61c856a69da7/kube-rbac-proxy/0.log" Sep 17 21:39:50.576124 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:50.576103 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsw4z_1b6918ad-3c4a-445e-932a-61c856a69da7/exporter/0.log" Sep 17 21:39:50.604325 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:50.604305 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsw4z_1b6918ad-3c4a-445e-932a-61c856a69da7/extractor/0.log" Sep 17 21:39:52.696174 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:52.696143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-44q44/perf-node-gather-daemonset-h9ttp" Sep 17 21:39:53.167610 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:53.167578 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8xmb9_b6753aa2-ef20-4371-9b8b-79db88916230/s3-init/0.log" Sep 17 21:39:53.183694 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:53.183669 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8xmb9_b6753aa2-ef20-4371-9b8b-79db88916230/download-hf-model/0.log" Sep 17 21:39:53.210536 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:53.210506 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-s6sx2_519ffa1d-20f5-452b-ac97-4346af6f8255/seaweedfs/0.log" Sep 17 21:39:59.285713 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.285687 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/kube-multus-additional-cni-plugins/0.log" Sep 17 21:39:59.308234 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.308213 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/egress-router-binary-copy/0.log" Sep 17 21:39:59.328410 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.328388 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/cni-plugins/0.log" Sep 17 21:39:59.347882 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.347859 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/bond-cni-plugin/0.log" Sep 17 21:39:59.369375 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.369353 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/routeoverride-cni/0.log" Sep 17 21:39:59.388029 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.388009 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/whereabouts-cni-bincopy/0.log" Sep 17 21:39:59.409427 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.409410 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t72xc_e434119f-e83f-428a-9061-2e131afb5f4f/whereabouts-cni/0.log" Sep 17 21:39:59.464952 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.464921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rtwxb_81833eee-01cf-420e-9cfb-8b6705c2db18/kube-multus/0.log" Sep 17 21:39:59.484969 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.484946 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f8sff_e85d8f30-029c-4596-a39d-101227da18aa/network-metrics-daemon/0.log" Sep 17 21:39:59.507985 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:39:59.507971 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f8sff_e85d8f30-029c-4596-a39d-101227da18aa/kube-rbac-proxy/0.log" Sep 17 21:40:00.545309 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.545270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/ovn-controller/0.log" Sep 17 21:40:00.566262 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.566237 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/ovn-acl-logging/0.log" Sep 17 21:40:00.589631 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.589610 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/kube-rbac-proxy-node/0.log" Sep 17 21:40:00.613700 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.613683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/kube-rbac-proxy-ovn-metrics/0.log" Sep 17 21:40:00.633248 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.633226 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/northd/0.log" Sep 17 21:40:00.653023 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.653002 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/nbdb/0.log" Sep 17 21:40:00.672974 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.672955 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/sbdb/0.log" Sep 17 21:40:00.710519 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:00.710497 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mnbqp_f662058f-fd45-4280-85de-0547407d6805/ovnkube-controller/0.log" Sep 17 21:40:01.904992 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:01.904951 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-mzsbm_b978e1d5-6c80-44f7-ae67-c8c3329d7b7c/check-endpoints/0.log" Sep 17 21:40:01.973120 ip-10-0-140-118 kubenswrapper[2579]: I0917 21:40:01.973085 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-z556m_90b744c7-94c6-400f-91eb-def724247bc0/network-check-target-container/0.log"