Jul 13 18:48:25.830623 ip-10-0-128-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 13 18:48:25.830634 ip-10-0-128-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 13 18:48:25.830642 ip-10-0-128-27 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 13 18:48:25.830955 ip-10-0-128-27 systemd[1]: Failed to start Kubernetes Kubelet. Jul 13 18:48:37.457558 ip-10-0-128-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 13 18:48:37.457576 ip-10-0-128-27 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6bbdfbba394e46449f4ca68c3878c605 -- Jul 13 18:50:10.053499 ip-10-0-128-27 systemd[1]: Starting Kubernetes Kubelet... Jul 13 18:50:10.518059 ip-10-0-128-27 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 18:50:10.518059 ip-10-0-128-27 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 18:50:10.518059 ip-10-0-128-27 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 18:50:10.518059 ip-10-0-128-27 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 18:50:10.518059 ip-10-0-128-27 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 18:50:10.518059 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.513319 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518169 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518181 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518185 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518188 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518191 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518194 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518197 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518201 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518204 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518206 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518210 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518213 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518215 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518218 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518221 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518223 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518226 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518230 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518233 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 18:50:10.519856 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518235 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518254 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518257 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518260 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518262 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518265 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518268 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518271 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518273 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518275 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518279 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518282 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518285 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518288 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518290 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518293 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518295 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518298 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518301 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518303 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 18:50:10.522031 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518306 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518309 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518311 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518315 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518317 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518320 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518322 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518327 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518335 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518338 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518342 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518345 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518348 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518351 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518354 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518358 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518361 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518364 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518367 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518370 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 18:50:10.523775 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518372 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518375 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518380 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518384 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518387 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518389 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518392 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518395 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518398 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518401 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518403 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518406 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518409 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518411 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518414 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518417 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518419 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518422 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518425 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 18:50:10.525916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518430 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518433 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518441 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518444 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518446 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518449 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518452 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518454 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518457 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518459 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518462 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518464 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518467 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518469 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518472 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518474 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518477 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518479 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518482 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518484 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 18:50:10.527745 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.518487 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519402 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519411 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519416 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519420 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519424 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519429 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519433 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519437 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519441 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519445 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519449 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519457 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519460 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519464 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519468 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519473 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519477 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519480 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519484 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 18:50:10.530137 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519488 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519492 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519496 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519500 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519504 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519509 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519514 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519517 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519521 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519525 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519529 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519533 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519537 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519541 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519544 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519549 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519554 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519558 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519562 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 18:50:10.545034 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519566 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519571 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519575 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519580 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519584 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519589 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519594 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519598 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519602 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519607 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519611 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519616 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519620 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519624 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519627 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519631 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519635 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519638 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519644 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 18:50:10.547151 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519648 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519652 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519655 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519659 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519663 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519666 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519670 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519674 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519677 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519681 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519685 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519688 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519692 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519696 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519700 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519703 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519707 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519711 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519717 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 18:50:10.548940 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519722 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519726 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519730 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519733 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519741 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519744 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519750 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519754 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519757 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519761 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519766 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519770 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519776 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519780 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519784 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519795 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519799 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519803 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519808 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519812 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 18:50:10.561563 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519816 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519820 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.519824 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520169 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520174 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520177 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520179 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520182 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520185 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520187 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520190 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520194 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520196 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520199 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520202 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520204 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520207 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520210 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520212 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 18:50:10.563533 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520215 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520220 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520223 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520226 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520229 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520233 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520236 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520253 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520255 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520258 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520260 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520263 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520266 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520268 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520271 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520274 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520277 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520281 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520284 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 18:50:10.565646 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520287 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520290 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520293 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520295 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520298 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520303 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520305 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520308 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520311 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520313 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520316 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520319 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520321 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520324 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520327 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520329 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520332 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520334 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520339 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520341 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 18:50:10.581013 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520344 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520346 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520349 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520352 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520354 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520357 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520359 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520362 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520365 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520368 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520371 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520374 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520378 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520381 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520384 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520386 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520389 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520393 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520395 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 18:50:10.583297 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520399 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520401 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520403 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520406 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520409 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520412 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520415 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520417 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520420 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520422 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520425 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520433 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520437 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520441 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520444 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520448 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520452 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520456 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520458 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520461 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 18:50:10.585129 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520465 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520468 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520472 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520476 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.520480 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520562 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520571 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520581 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520587 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520593 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520599 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520604 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520610 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520614 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520619 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520624 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520628 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520633 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520639 2560 flags.go:64] FLAG: --cgroup-root="" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520643 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520648 2560 flags.go:64] FLAG: --client-ca-file="" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520652 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520656 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520663 2560 flags.go:64] FLAG: --cluster-domain="" Jul 13 18:50:10.597520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520668 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 18:50:10.597785 ip-10-0-128-27 systemd[1]: Started Kubernetes Kubelet. Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520673 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520677 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520682 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520687 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520691 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520694 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520698 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520702 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520707 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520711 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520716 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520720 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520725 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520730 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520734 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520738 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520742 2560 flags.go:64] FLAG: --enable-server="true" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520747 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520754 2560 flags.go:64] FLAG: --event-burst="100" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520758 2560 flags.go:64] FLAG: --event-qps="50" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520762 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520766 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520770 2560 flags.go:64] FLAG: --eviction-hard="" Jul 13 18:50:10.600190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520776 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520780 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520784 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520789 2560 flags.go:64] FLAG: --eviction-soft="" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520793 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520797 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520801 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520807 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520811 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520815 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520819 2560 flags.go:64] FLAG: --feature-gates="" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520824 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520828 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520832 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520837 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520841 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520845 2560 flags.go:64] FLAG: --help="false" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520849 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-128-27.ec2.internal" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520854 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520859 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520863 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520868 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520873 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520877 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 18:50:10.602983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520881 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520885 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520891 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520896 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520899 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520902 2560 flags.go:64] FLAG: --kube-reserved="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520906 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520911 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520915 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520919 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520923 2560 flags.go:64] FLAG: --lock-file="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520927 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520932 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520936 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520943 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520948 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520953 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520957 2560 flags.go:64] FLAG: --logging-format="text" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520961 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520966 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520970 2560 flags.go:64] FLAG: --manifest-url="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520975 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520980 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520984 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520990 2560 flags.go:64] FLAG: --max-pods="110" Jul 13 18:50:10.842521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520994 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.520998 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521002 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521006 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521011 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521015 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521019 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521028 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521033 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521039 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521043 2560 flags.go:64] FLAG: --pod-cidr="" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521047 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521055 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521059 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521064 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521068 2560 flags.go:64] FLAG: --port="10250" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521072 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521077 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0daa93caf466b287e" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521082 2560 flags.go:64] FLAG: --qos-reserved="" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521086 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521091 2560 flags.go:64] FLAG: --register-node="true" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521095 2560 flags.go:64] FLAG: --register-with-taints="" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521101 2560 flags.go:64] FLAG: --registry-burst="10" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521104 2560 flags.go:64] FLAG: --registry-qps="5" Jul 13 18:50:10.847367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521106 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521110 2560 flags.go:64] FLAG: --reserved-memory="" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521115 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521119 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521124 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521128 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521132 2560 flags.go:64] FLAG: --runonce="false" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521136 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521141 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521146 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521150 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521155 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521159 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521163 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521168 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521172 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521176 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521180 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521186 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521191 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521195 2560 flags.go:64] FLAG: --system-cgroups="" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521199 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521206 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521210 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521215 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 18:50:10.856465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521220 2560 flags.go:64] FLAG: --tls-min-version="" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521223 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521228 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521232 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521252 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521259 2560 flags.go:64] FLAG: --v="2" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521265 2560 flags.go:64] FLAG: --version="false" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521271 2560 flags.go:64] FLAG: --vmodule="" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521277 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.521282 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521471 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521481 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521485 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521491 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521497 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521502 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521506 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521510 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521514 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521517 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521521 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 18:50:10.857523 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521524 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521528 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521532 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521536 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521541 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521546 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521550 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521553 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521557 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521561 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521565 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521569 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521572 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521576 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521580 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521584 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521596 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521600 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521604 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521609 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 18:50:10.858414 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521612 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521616 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521620 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521623 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521629 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521634 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521638 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521641 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521645 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521648 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521652 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521656 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521659 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521663 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521666 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521670 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521676 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521680 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521684 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521687 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 18:50:10.859044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521691 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521696 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521699 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521704 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521708 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521711 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521715 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521719 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521725 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521735 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521739 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521742 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521746 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521750 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521754 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521757 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521761 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521764 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521768 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521772 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 18:50:10.859658 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521776 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521780 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521783 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521787 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521791 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521794 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521798 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521802 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521807 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521811 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521815 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521818 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521822 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521826 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521829 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521833 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521837 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521840 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521844 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521847 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 18:50:10.862422 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521852 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521856 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521867 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521871 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521875 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521878 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521882 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.521885 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.522566 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.531721 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.531735 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.531753 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.531759 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531835 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 18:50:10.863012 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531841 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531844 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531847 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531852 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531855 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531858 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531861 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531864 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531866 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531869 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531872 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531875 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531877 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531880 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531882 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531885 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531888 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531891 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531893 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 18:50:10.863638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531898 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531901 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531905 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531908 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531911 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531913 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531916 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531919 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531922 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531924 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531928 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531932 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531935 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531938 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531941 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531944 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531947 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531949 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531952 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531955 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 18:50:10.864251 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531958 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531961 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531963 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531966 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531969 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531972 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531974 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531977 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531980 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531982 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531985 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531988 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531991 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531993 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531996 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.531998 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532001 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532003 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532006 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532008 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 18:50:10.864916 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532011 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532013 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532016 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532019 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532022 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532024 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532027 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532029 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532032 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532034 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532037 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532039 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532041 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532044 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532046 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532049 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532051 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532054 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532056 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532059 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 18:50:10.865477 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532062 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532064 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532067 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532070 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532072 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532075 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532077 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532080 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532083 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532085 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532088 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532091 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532093 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532096 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532098 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532101 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532103 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532106 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 18:50:10.866044 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532109 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.532113 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532221 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532226 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532229 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532232 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532235 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532256 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532261 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532264 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532267 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532269 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532272 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532275 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532278 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 18:50:10.866542 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532281 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532283 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532286 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532289 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532291 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532294 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532296 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532299 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532302 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532304 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532307 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532309 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532312 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532315 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532317 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532320 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532323 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532327 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532331 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 18:50:10.866990 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532333 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532336 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532339 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532342 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532344 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532347 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532349 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532352 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532354 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532357 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532359 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532362 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532364 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532367 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532370 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532372 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532375 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532377 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532380 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532382 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 18:50:10.867541 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532385 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532388 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532390 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532394 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532397 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532400 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532403 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532406 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532410 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532413 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532416 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532419 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532422 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532424 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532427 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532429 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532432 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532435 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532437 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 18:50:10.868122 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532440 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532442 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532445 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532448 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532450 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532453 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532455 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532458 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532460 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532463 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532465 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532467 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532470 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532472 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532475 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532477 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532479 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532482 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532484 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532487 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 18:50:10.868655 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532489 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532492 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532494 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532497 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532499 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532501 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532507 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:10.532510 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.532515 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.532609 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.534996 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.537842 2560 server.go:1019] "Starting client certificate rotation" Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.537939 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 18:50:11.032624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.537974 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.540669 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.543713 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.556218 2560 log.go:25] "Validated CRI v1 runtime API" Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.561563 2560 log.go:25] "Validated CRI v1 image API" Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.563569 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.566721 2560 fs.go:135] Filesystem UUIDs: map[318d4520-2d56-4572-977d-63f7c852f86b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8e8ce701-5df8-46be-8356-a88cff08624b:/dev/nvme0n1p3] Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.566738 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 18:50:11.033180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.568540 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.574049 2560 manager.go:216] Machine: {Timestamp:2026-07-13 18:50:10.571783367 +0000 UTC m=+0.380489359 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096565 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac21ae4fc84c43e65e7106acaff69 SystemUUID:ec2ac21a-e4fc-84c4-3e65-e7106acaff69 BootID:6bbdfbba-394e-4644-9f4c-a68c3878c605 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:16:35:60:fd:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:16:35:60:fd:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:28:52:1a:5f:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.574141 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.574212 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.575985 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.576005 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.576840 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.576856 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.576864 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.576876 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.578801 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583558 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583700 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583727 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583739 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583749 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583757 2560 kubelet.go:398] "Adding apiserver pod source" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.583765 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 18:50:11.035031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.584426 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.584441 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.589484 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.591750 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.591778 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592587 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592601 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592607 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592613 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592621 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592629 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592676 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592682 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592689 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592694 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592702 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.592710 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.593581 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.593591 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.595357 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.596282 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.596291 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.596947 2560 server.go:1268] "Started kubelet" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.597010 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.597037 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.597137 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.598288 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.598517 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.604428 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607053 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607057 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.607151 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607811 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607812 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607845 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607955 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.607966 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608084 2560 factory.go:153] Registering CRI-O factory Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608122 2560 factory.go:223] Registration of the crio container factory successfully Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608183 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608198 2560 factory.go:55] Registering systemd factory Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608205 2560 factory.go:223] Registration of the systemd container factory successfully Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608219 2560 factory.go:103] Registering Raw factory Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608233 2560 manager.go:1196] Started watching for new ooms in manager Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.608682 2560 manager.go:349] Starting recovery of all containers Jul 13 18:50:11.036017 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.612463 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.612513 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.612531 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdd1396a6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.59691895 +0000 UTC m=+0.405624941,LastTimestamp:2026-07-13 18:50:10.59691895 +0000 UTC m=+0.405624941,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.618386 2560 manager.go:354] Recovery completed Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.626445 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.629784 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.629811 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.629821 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.630329 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.630341 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.630331 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.630359 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.631086 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.631109 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.631121 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.632541 2560 policy_none.go:49] "None policy: Start" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.632555 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.632567 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.632768 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf094976 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629798262 +0000 UTC m=+0.438504257,LastTimestamp:2026-07-13 18:50:10.629798262 +0000 UTC m=+0.438504257,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.634690 2560 policy_none.go:47] "Start" Jul 13 18:50:11.175923 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.642030 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf098e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629815936 +0000 UTC m=+0.438521929,LastTimestamp:2026-07-13 18:50:10.629815936 +0000 UTC m=+0.438521929,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.654014 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf09b168 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629824872 +0000 UTC m=+0.438530864,LastTimestamp:2026-07-13 18:50:10.629824872 +0000 UTC m=+0.438530864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.668791 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf094976\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf094976 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629798262 +0000 UTC m=+0.438504257,LastTimestamp:2026-07-13 18:50:10.631095 +0000 UTC m=+0.439800993,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.677273 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf098e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf098e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629815936 +0000 UTC m=+0.438521929,LastTimestamp:2026-07-13 18:50:10.631115783 +0000 UTC m=+0.439821775,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.683704 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.684934 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.684980 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.685000 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.685031 2560 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.688681 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf09b168\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf09b168 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629824872 +0000 UTC m=+0.438530864,LastTimestamp:2026-07-13 18:50:10.631124912 +0000 UTC m=+0.439830903,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.688733 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.698959 2560 manager.go:337] "Starting Device Plugin manager" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.698988 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.698998 2560 server.go:87] "Starting device plugin registration server" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.699206 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.699217 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.699328 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.699396 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 18:50:11.177349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.699405 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.699888 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.699934 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.718771 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffe4367cca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.716646602 +0000 UTC m=+0.525352582,LastTimestamp:2026-07-13 18:50:10.716646602 +0000 UTC m=+0.525352582,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.728523 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bc2l5" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.739484 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bc2l5" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.786039 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal"] Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.786140 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.787019 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.787046 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.787056 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.795661 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf094976\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf094976 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629798262 +0000 UTC m=+0.438504257,LastTimestamp:2026-07-13 18:50:10.787032505 +0000 UTC m=+0.595738505,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.800329 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.801021 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.801046 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.801056 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.801077 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.806948 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf098e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf098e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629815936 +0000 UTC m=+0.438521929,LastTimestamp:2026-07-13 18:50:10.787051206 +0000 UTC m=+0.595757197,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.177901 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.813927 2560 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-128-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.813967 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf09b168\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf09b168 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629824872 +0000 UTC m=+0.438530864,LastTimestamp:2026-07-13 18:50:10.787060439 +0000 UTC m=+0.595766430,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817168 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817277 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817308 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817884 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817886 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817906 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817921 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.817924 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817937 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.817922 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.817890 2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-27.ec2.internal.18c1edffdf094976\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-27.ec2.internal.18c1edffdf094976 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-27.ec2.internal,UID:ip-10-0-128-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-27.ec2.internal,},FirstTimestamp:2026-07-13 18:50:10.629798262 +0000 UTC m=+0.438504257,LastTimestamp:2026-07-13 18:50:10.801032401 +0000 UTC m=+0.609738393,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-27.ec2.internal,}" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.820134 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.820154 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.820753 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.820773 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.820784 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.854740 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-27.ec2.internal\" not found" node="ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:10.859412 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-27.ec2.internal\" not found" node="ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.909571 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a0e4fc61c3258bb0daab266fcb1a4740-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal\" (UID: \"a0e4fc61c3258bb0daab266fcb1a4740\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.178482 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.910004 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e4fc61c3258bb0daab266fcb1a4740-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal\" (UID: \"a0e4fc61c3258bb0daab266fcb1a4740\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:10.910021 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eed0f0b6c98d844417e6646ccde34c95-config\") pod \"kube-apiserver-proxy-ip-10-0-128-27.ec2.internal\" (UID: \"eed0f0b6c98d844417e6646ccde34c95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.010288 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eed0f0b6c98d844417e6646ccde34c95-config\") pod \"kube-apiserver-proxy-ip-10-0-128-27.ec2.internal\" (UID: \"eed0f0b6c98d844417e6646ccde34c95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.010312 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a0e4fc61c3258bb0daab266fcb1a4740-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal\" (UID: \"a0e4fc61c3258bb0daab266fcb1a4740\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.010331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e4fc61c3258bb0daab266fcb1a4740-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal\" (UID: \"a0e4fc61c3258bb0daab266fcb1a4740\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.010358 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e4fc61c3258bb0daab266fcb1a4740-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal\" (UID: \"a0e4fc61c3258bb0daab266fcb1a4740\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.010375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eed0f0b6c98d844417e6646ccde34c95-config\") pod \"kube-apiserver-proxy-ip-10-0-128-27.ec2.internal\" (UID: \"eed0f0b6c98d844417e6646ccde34c95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.010388 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a0e4fc61c3258bb0daab266fcb1a4740-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal\" (UID: \"a0e4fc61c3258bb0daab266fcb1a4740\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.014380 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.017792 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.017819 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.017830 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.017858 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.028079 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.028100 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-27.ec2.internal\": node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.053044 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.153547 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.157714 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.179073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.163278 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" Jul 13 18:50:11.346423 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.254427 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.507576 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.354736 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.507576 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.455192 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.536827 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.555335 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.656264 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.682510 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.684138 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientMemory" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.684168 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 18:50:11.684365 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.684179 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeHasSufficientPID" Jul 13 18:50:11.868048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.725635 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 18:50:11.868048 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:11.756538 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-27.ec2.internal\" not found" Jul 13 18:50:11.868048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.757164 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 18:50:11.868048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.819911 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 18:45:11 +0000 UTC" deadline="2027-12-13 12:52:26.146020467 +0000 UTC" Jul 13 18:50:11.868048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.819956 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12426h2m14.326069165s" Jul 13 18:50:11.868048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.829484 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 18:50:11.872580 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:11.872157 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e4fc61c3258bb0daab266fcb1a4740.slice/crio-a12f6d08bf9abf0cf8c9cadc1fea9707f4e19fd0dd495d7c6055d84a0720fd22 WatchSource:0}: Error finding container a12f6d08bf9abf0cf8c9cadc1fea9707f4e19fd0dd495d7c6055d84a0720fd22: Status 404 returned error can't find the container with id a12f6d08bf9abf0cf8c9cadc1fea9707f4e19fd0dd495d7c6055d84a0720fd22 Jul 13 18:50:12.090096 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.875323 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 18:50:12.090096 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.879255 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 18:50:12.090096 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.908943 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" Jul 13 18:50:12.090096 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.926264 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 18:50:12.090096 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.927522 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" Jul 13 18:50:12.090096 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:11.935531 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 18:50:12.247110 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.180182 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed0f0b6c98d844417e6646ccde34c95.slice/crio-c182ac59acf19c45d7d9e00690e856fd2e0b385b430fb4b5deadf0c615e991a0 WatchSource:0}: Error finding container c182ac59acf19c45d7d9e00690e856fd2e0b385b430fb4b5deadf0c615e991a0: Status 404 returned error can't find the container with id c182ac59acf19c45d7d9e00690e856fd2e0b385b430fb4b5deadf0c615e991a0 Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.585979 2560 apiserver.go:52] "Watching apiserver" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.594538 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.595333 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6mvdw","openshift-network-diagnostics/network-check-target-2zwjj","kube-system/global-pull-secret-syncer-x6lrk","kube-system/konnectivity-agent-zzhs5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr","openshift-cluster-node-tuning-operator/tuned-2hmf5","openshift-image-registry/node-ca-trm8k","openshift-multus/multus-ghf6p","openshift-network-operator/iptables-alerter-9gxff","openshift-ovn-kubernetes/ovnkube-node-k9k97","kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal","openshift-dns/node-resolver-9pjjf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal","openshift-multus/multus-additional-cni-plugins-vlhmn"] Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.599402 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.601971 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.602463 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vv6cw\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.602612 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.603291 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.603542 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.604030 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.605362 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.605599 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.605718 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rtbkd\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.606367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607100 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607337 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607489 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gvlgg\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607508 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607519 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607531 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.607768 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.608770 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.609144 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.609290 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mv8w6\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.609349 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.611010 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.611596 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.611629 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fn6nj\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.611952 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.611961 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.613390 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.614518 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.614606 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.614767 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w4p8m\"" Jul 13 18:50:12.706213 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.616166 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.616243 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.616415 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hqftg\"" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.616545 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.617042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.617138 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619408 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmgc\" (UniqueName: \"kubernetes.io/projected/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-kube-api-access-cjmgc\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43347716-0baa-494f-9649-aeea5a817ac6-tmp-dir\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619559 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmsz\" (UniqueName: \"kubernetes.io/projected/43347716-0baa-494f-9649-aeea5a817ac6-kube-api-access-mqmsz\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619610 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-system-cni-dir\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619645 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-cnibin\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619668 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619739 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62eaf7ee-ca92-4910-800b-e59a57f2776e-agent-certs\") pod \"konnectivity-agent-zzhs5\" (UID: \"62eaf7ee-ca92-4910-800b-e59a57f2776e\") " pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-host\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-os-release\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-socket-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-registration-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619953 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619953 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 18:50:12.707839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.619955 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-device-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.620067 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5x24\" (UniqueName: \"kubernetes.io/projected/89785005-2d32-4f4b-9e9d-c52feaf07769-kube-api-access-q5x24\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.621769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.621819 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.621859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-serviceca\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.621897 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnm8r\" (UniqueName: \"kubernetes.io/projected/774de39e-107f-49e0-9aca-8efc914d41ee-kube-api-access-vnm8r\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.621946 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62eaf7ee-ca92-4910-800b-e59a57f2776e-konnectivity-ca\") pod \"konnectivity-agent-zzhs5\" (UID: \"62eaf7ee-ca92-4910-800b-e59a57f2776e\") " pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.621985 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43347716-0baa-494f-9649-aeea5a817ac6-hosts-file\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.622038 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.622063 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.622118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-etc-selinux\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.622178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-sys-fs\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.623325 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.624063 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.624175 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dzjz4\"" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.625909 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.625966 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.626138 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.626261 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.626303 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.626322 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 18:50:12.708551 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.627358 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.627414 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.627424 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4d59l\"" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.628455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.628499 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.654697 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gljkh" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.663803 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gljkh" Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.689187 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" event={"ID":"eed0f0b6c98d844417e6646ccde34c95","Type":"ContainerStarted","Data":"c182ac59acf19c45d7d9e00690e856fd2e0b385b430fb4b5deadf0c615e991a0"} Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.690302 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" event={"ID":"a0e4fc61c3258bb0daab266fcb1a4740","Type":"ContainerStarted","Data":"a12f6d08bf9abf0cf8c9cadc1fea9707f4e19fd0dd495d7c6055d84a0720fd22"} Jul 13 18:50:12.709336 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.708574 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-systemd\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-cni-netd\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723246 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-socket-dir-parent\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723285 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-cni-bin\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723360 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-daemon-config\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723385 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysconfig\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723438 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-env-overrides\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-ovnkube-script-lib\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723491 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc5ddbf7-93cd-498b-b808-de40d5c7065b-iptables-alerter-script\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-netns\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723532 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723547 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-sys\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723567 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723584 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-var-lib-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723614 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-multus-certs\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.864865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723638 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbfmw\" (UniqueName: \"kubernetes.io/projected/bac567a5-a97c-4a54-97c7-946025cc3b08-kube-api-access-kbfmw\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723721 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-run-netns\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-ovn\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723752 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723775 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62eaf7ee-ca92-4910-800b-e59a57f2776e-agent-certs\") pod \"konnectivity-agent-zzhs5\" (UID: \"62eaf7ee-ca92-4910-800b-e59a57f2776e\") " pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723823 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-etc-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-hostroot\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723869 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-kubernetes\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723947 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-run\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.723981 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-var-lib-kubelet\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-socket-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724036 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-kubelet\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-systemd-units\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724085 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-system-cni-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724119 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-socket-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724123 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-cni-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.865747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724151 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724172 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-serviceca\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724158 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724205 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-node-log\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724236 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-os-release\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724273 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-cni-multus\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724297 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-systemd\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724316 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bac567a5-a97c-4a54-97c7-946025cc3b08-tmp\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724350 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnm9x\" (UniqueName: \"kubernetes.io/projected/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-kube-api-access-dnm9x\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724388 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-tuned\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w25fm\" (UniqueName: \"kubernetes.io/projected/818910c4-05c0-416f-a7cc-4feee88e85c8-kube-api-access-w25fm\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724442 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-device-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724487 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-device-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-serviceca\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724517 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724517 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmgc\" (UniqueName: \"kubernetes.io/projected/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-kube-api-access-cjmgc\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.866752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724562 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43347716-0baa-494f-9649-aeea5a817ac6-tmp-dir\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-cnibin\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724600 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724620 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysctl-d\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724638 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-slash\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724653 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724678 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-modprobe-d\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724710 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-cnibin\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm8r\" (UniqueName: \"kubernetes.io/projected/774de39e-107f-49e0-9aca-8efc914d41ee-kube-api-access-vnm8r\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62eaf7ee-ca92-4910-800b-e59a57f2776e-konnectivity-ca\") pod \"konnectivity-agent-zzhs5\" (UID: \"62eaf7ee-ca92-4910-800b-e59a57f2776e\") " pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724812 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43347716-0baa-494f-9649-aeea5a817ac6-tmp-dir\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-log-socket\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724967 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-ovnkube-config\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.724991 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-cni-binary-copy\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725013 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-conf-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-sys-fs\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725089 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.867612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725111 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725123 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725115 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-sys-fs\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-cnibin\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725177 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-k8s-cni-cncf-io\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f21d64c2-eeba-4c25-92df-a13d3968ac4a-dbus\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725229 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5x24\" (UniqueName: \"kubernetes.io/projected/89785005-2d32-4f4b-9e9d-c52feaf07769-kube-api-access-q5x24\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725272 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmsz\" (UniqueName: \"kubernetes.io/projected/43347716-0baa-494f-9649-aeea5a817ac6-kube-api-access-mqmsz\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725298 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f21d64c2-eeba-4c25-92df-a13d3968ac4a-kubelet-config\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-host\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725340 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62eaf7ee-ca92-4910-800b-e59a57f2776e-konnectivity-ca\") pod \"konnectivity-agent-zzhs5\" (UID: \"62eaf7ee-ca92-4910-800b-e59a57f2776e\") " pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725349 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-os-release\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-registration-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725398 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725405 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-host\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725420 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-cni-bin\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725494 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-registration-dir\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.868483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725541 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-os-release\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/bc5ddbf7-93cd-498b-b808-de40d5c7065b-kube-api-access-h826z\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725592 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725640 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-etc-kubernetes\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725657 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysctl-conf\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725680 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-lib-modules\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-host\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43347716-0baa-494f-9649-aeea5a817ac6-hosts-file\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725782 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43347716-0baa-494f-9649-aeea5a817ac6-hosts-file\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725785 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-etc-selinux\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725811 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/818910c4-05c0-416f-a7cc-4feee88e85c8-ovn-node-metrics-cert\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725835 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5ddbf7-93cd-498b-b808-de40d5c7065b-host-slash\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725862 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89785005-2d32-4f4b-9e9d-c52feaf07769-etc-selinux\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725878 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-kubelet\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725902 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2rd\" (UniqueName: \"kubernetes.io/projected/17b4d31c-1830-4dc8-8984-9032c38bcc3e-kube-api-access-8q2rd\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:12.869358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.725950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-system-cni-dir\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.726020 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/774de39e-107f-49e0-9aca-8efc914d41ee-system-cni-dir\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.726084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/774de39e-107f-49e0-9aca-8efc914d41ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.728386 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62eaf7ee-ca92-4910-800b-e59a57f2776e-agent-certs\") pod \"konnectivity-agent-zzhs5\" (UID: \"62eaf7ee-ca92-4910-800b-e59a57f2776e\") " pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.734074 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmgc\" (UniqueName: \"kubernetes.io/projected/7b6dacaf-10a8-49ca-bfeb-1ddf56986535-kube-api-access-cjmgc\") pod \"node-ca-trm8k\" (UID: \"7b6dacaf-10a8-49ca-bfeb-1ddf56986535\") " pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.734112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmsz\" (UniqueName: \"kubernetes.io/projected/43347716-0baa-494f-9649-aeea5a817ac6-kube-api-access-mqmsz\") pod \"node-resolver-9pjjf\" (UID: \"43347716-0baa-494f-9649-aeea5a817ac6\") " pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.734263 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5x24\" (UniqueName: \"kubernetes.io/projected/89785005-2d32-4f4b-9e9d-c52feaf07769-kube-api-access-q5x24\") pod \"aws-ebs-csi-driver-node-t49mr\" (UID: \"89785005-2d32-4f4b-9e9d-c52feaf07769\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.734416 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnm8r\" (UniqueName: \"kubernetes.io/projected/774de39e-107f-49e0-9aca-8efc914d41ee-kube-api-access-vnm8r\") pod \"multus-additional-cni-plugins-vlhmn\" (UID: \"774de39e-107f-49e0-9aca-8efc914d41ee\") " pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827223 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-cnibin\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-k8s-cni-cncf-io\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827345 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-cnibin\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827362 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f21d64c2-eeba-4c25-92df-a13d3968ac4a-dbus\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f21d64c2-eeba-4c25-92df-a13d3968ac4a-kubelet-config\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827411 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-k8s-cni-cncf-io\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827461 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-cni-bin\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.827472 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:12.871757 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/bc5ddbf7-93cd-498b-b808-de40d5c7065b-kube-api-access-h826z\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-etc-kubernetes\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827519 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f21d64c2-eeba-4c25-92df-a13d3968ac4a-kubelet-config\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.827537 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:13.327516575 +0000 UTC m=+3.136222571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827545 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-etc-kubernetes\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827560 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysctl-conf\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827587 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-lib-modules\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827608 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-host\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827620 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-cni-bin\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827625 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f21d64c2-eeba-4c25-92df-a13d3968ac4a-dbus\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/818910c4-05c0-416f-a7cc-4feee88e85c8-ovn-node-metrics-cert\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827660 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5ddbf7-93cd-498b-b808-de40d5c7065b-host-slash\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827683 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-kubelet\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-lib-modules\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827729 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2rd\" (UniqueName: \"kubernetes.io/projected/17b4d31c-1830-4dc8-8984-9032c38bcc3e-kube-api-access-8q2rd\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:12.872663 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-systemd\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827766 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-host\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827776 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-cni-netd\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827793 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc5ddbf7-93cd-498b-b808-de40d5c7065b-host-slash\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysctl-conf\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-socket-dir-parent\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827848 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-socket-dir-parent\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827867 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-cni-bin\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827869 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-kubelet\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-systemd\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827891 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-daemon-config\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827915 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-cni-bin\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827920 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysconfig\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827945 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-env-overrides\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827962 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-cni-netd\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.827967 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-ovnkube-script-lib\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc5ddbf7-93cd-498b-b808-de40d5c7065b-iptables-alerter-script\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.828029 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:12.873604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828062 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-netns\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.828083 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:13.328063805 +0000 UTC m=+3.136769801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828030 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-netns\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828150 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-sys\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828175 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-var-lib-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828201 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-multus-certs\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbfmw\" (UniqueName: \"kubernetes.io/projected/bac567a5-a97c-4a54-97c7-946025cc3b08-kube-api-access-kbfmw\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-run-netns\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828274 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-ovn\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828300 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-etc-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828342 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-hostroot\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-kubernetes\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-run\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828418 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-var-lib-kubelet\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828443 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-env-overrides\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-sys\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828554 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-var-lib-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828597 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-run-multus-certs\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-kubelet\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-systemd-units\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-system-cni-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc5ddbf7-93cd-498b-b808-de40d5c7065b-iptables-alerter-script\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828762 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-cni-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828696 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-cni-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828778 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-kubelet\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828801 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-run-netns\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-node-log\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828827 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-systemd-units\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828836 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-node-log\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-os-release\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828872 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-cni-multus\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-system-cni-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828885 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-ovn\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828900 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-systemd\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.875946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828914 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-hostroot\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828952 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysconfig\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-host-var-lib-cni-multus\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828984 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bac567a5-a97c-4a54-97c7-946025cc3b08-tmp\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-run\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829005 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-os-release\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829014 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnm9x\" (UniqueName: \"kubernetes.io/projected/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-kube-api-access-dnm9x\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829043 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-kubernetes\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829083 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-systemd\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829091 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-ovnkube-script-lib\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829117 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-etc-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.828958 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-var-lib-kubelet\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829096 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-tuned\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829164 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-daemon-config\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w25fm\" (UniqueName: \"kubernetes.io/projected/818910c4-05c0-416f-a7cc-4feee88e85c8-kube-api-access-w25fm\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysctl-d\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-slash\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.876879 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829523 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-modprobe-d\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-log-socket\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-ovnkube-config\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-sysctl-d\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829629 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-cni-binary-copy\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-conf-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829783 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-host-slash\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.829812 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-run-openvswitch\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.830082 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/818910c4-05c0-416f-a7cc-4feee88e85c8-ovnkube-config\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.830192 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-modprobe-d\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.830237 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/818910c4-05c0-416f-a7cc-4feee88e85c8-log-socket\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.830270 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-cni-binary-copy\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.830305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-multus-conf-dir\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.831602 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bac567a5-a97c-4a54-97c7-946025cc3b08-tmp\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.831843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/818910c4-05c0-416f-a7cc-4feee88e85c8-ovn-node-metrics-cert\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.832271 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bac567a5-a97c-4a54-97c7-946025cc3b08-etc-tuned\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.836620 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:12.878050 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.836640 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.836652 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:12.836722 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:13.336707426 +0000 UTC m=+3.145413425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.839342 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnm9x\" (UniqueName: \"kubernetes.io/projected/bcc88ca5-0a67-4ea4-a976-4d1f1a86f677-kube-api-access-dnm9x\") pod \"multus-ghf6p\" (UID: \"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677\") " pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.839587 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbfmw\" (UniqueName: \"kubernetes.io/projected/bac567a5-a97c-4a54-97c7-946025cc3b08-kube-api-access-kbfmw\") pod \"tuned-2hmf5\" (UID: \"bac567a5-a97c-4a54-97c7-946025cc3b08\") " pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.839734 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2rd\" (UniqueName: \"kubernetes.io/projected/17b4d31c-1830-4dc8-8984-9032c38bcc3e-kube-api-access-8q2rd\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.840089 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w25fm\" (UniqueName: \"kubernetes.io/projected/818910c4-05c0-416f-a7cc-4feee88e85c8-kube-api-access-w25fm\") pod \"ovnkube-node-k9k97\" (UID: \"818910c4-05c0-416f-a7cc-4feee88e85c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.878975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.841260 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/bc5ddbf7-93cd-498b-b808-de40d5c7065b-kube-api-access-h826z\") pod \"iptables-alerter-9gxff\" (UID: \"bc5ddbf7-93cd-498b-b808-de40d5c7065b\") " pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.911273 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-trm8k" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.918735 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9pjjf" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.928720 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.931524 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.937435 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.943119 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.949687 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6dacaf_10a8_49ca_bfeb_1ddf56986535.slice/crio-ab1471840105735fa97f7e19406406232c8b8d54b3fd943786262adb3c603b20 WatchSource:0}: Error finding container ab1471840105735fa97f7e19406406232c8b8d54b3fd943786262adb3c603b20: Status 404 returned error can't find the container with id ab1471840105735fa97f7e19406406232c8b8d54b3fd943786262adb3c603b20 Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.954916 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gxff" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.960163 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ghf6p" Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.964428 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43347716_0baa_494f_9649_aeea5a817ac6.slice/crio-6892ecf686fd23a69eb5290c3f0c3d1bf59dba7a45bb0d9ad77b50b63a124f69 WatchSource:0}: Error finding container 6892ecf686fd23a69eb5290c3f0c3d1bf59dba7a45bb0d9ad77b50b63a124f69: Status 404 returned error can't find the container with id 6892ecf686fd23a69eb5290c3f0c3d1bf59dba7a45bb0d9ad77b50b63a124f69 Jul 13 18:50:12.973974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:12.966132 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:12.977183 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.975299 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac567a5_a97c_4a54_97c7_946025cc3b08.slice/crio-9af61727ef13126cf1a15468ad3eafe0db10833bfdb7f40dfadf4e81063a48e3 WatchSource:0}: Error finding container 9af61727ef13126cf1a15468ad3eafe0db10833bfdb7f40dfadf4e81063a48e3: Status 404 returned error can't find the container with id 9af61727ef13126cf1a15468ad3eafe0db10833bfdb7f40dfadf4e81063a48e3 Jul 13 18:50:12.977183 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.976086 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eaf7ee_ca92_4910_800b_e59a57f2776e.slice/crio-9bc2aacfbca41a267c04c8b58479a3b9086d29ff0c090198bfa956cb0e2f0ba0 WatchSource:0}: Error finding container 9bc2aacfbca41a267c04c8b58479a3b9086d29ff0c090198bfa956cb0e2f0ba0: Status 404 returned error can't find the container with id 9bc2aacfbca41a267c04c8b58479a3b9086d29ff0c090198bfa956cb0e2f0ba0 Jul 13 18:50:12.988559 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.979443 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89785005_2d32_4f4b_9e9d_c52feaf07769.slice/crio-58ae77857b4afe8180dad5c557ae6072e55c3b0b0a638363318eb586fed38214 WatchSource:0}: Error finding container 58ae77857b4afe8180dad5c557ae6072e55c3b0b0a638363318eb586fed38214: Status 404 returned error can't find the container with id 58ae77857b4afe8180dad5c557ae6072e55c3b0b0a638363318eb586fed38214 Jul 13 18:50:12.988559 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.984709 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5ddbf7_93cd_498b_b808_de40d5c7065b.slice/crio-371f9fc0b50e9fec9d139bb2b01e8ac38ca6a26ce7f6749d168b97874ad0f3dd WatchSource:0}: Error finding container 371f9fc0b50e9fec9d139bb2b01e8ac38ca6a26ce7f6749d168b97874ad0f3dd: Status 404 returned error can't find the container with id 371f9fc0b50e9fec9d139bb2b01e8ac38ca6a26ce7f6749d168b97874ad0f3dd Jul 13 18:50:12.988559 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.987182 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc88ca5_0a67_4ea4_a976_4d1f1a86f677.slice/crio-0dc75540206be70e86100389131b12573a4c24183da04531efe1446114c6caee WatchSource:0}: Error finding container 0dc75540206be70e86100389131b12573a4c24183da04531efe1446114c6caee: Status 404 returned error can't find the container with id 0dc75540206be70e86100389131b12573a4c24183da04531efe1446114c6caee Jul 13 18:50:12.992536 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:12.991494 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818910c4_05c0_416f_a7cc_4feee88e85c8.slice/crio-65689ecefb6ab19e290de76d28675b61167d1cdf41037653ef8fd6a673a3def9 WatchSource:0}: Error finding container 65689ecefb6ab19e290de76d28675b61167d1cdf41037653ef8fd6a673a3def9: Status 404 returned error can't find the container with id 65689ecefb6ab19e290de76d28675b61167d1cdf41037653ef8fd6a673a3def9 Jul 13 18:50:13.390986 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.334219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:13.390986 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.334307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:13.390986 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.334376 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:13.390986 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.334465 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:14.334444446 +0000 UTC m=+4.143150429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:13.390986 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.334469 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:13.390986 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.334522 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:14.33450555 +0000 UTC m=+4.143211549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:13.435539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.435459 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:13.435681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.435656 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:13.435681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.435681 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:13.435801 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.435695 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:13.435801 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:13.435763 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:14.435743232 +0000 UTC m=+4.244449461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:13.667056 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.666982 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:13 +0000 UTC" deadline="2027-12-31 11:51:00.93235947 +0000 UTC" Jul 13 18:50:13.667056 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.667054 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12857h0m47.265309485s" Jul 13 18:50:13.695641 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.695561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ghf6p" event={"ID":"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677","Type":"ContainerStarted","Data":"0dc75540206be70e86100389131b12573a4c24183da04531efe1446114c6caee"} Jul 13 18:50:13.697971 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.697937 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" event={"ID":"89785005-2d32-4f4b-9e9d-c52feaf07769","Type":"ContainerStarted","Data":"58ae77857b4afe8180dad5c557ae6072e55c3b0b0a638363318eb586fed38214"} Jul 13 18:50:13.700141 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.700114 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" event={"ID":"bac567a5-a97c-4a54-97c7-946025cc3b08","Type":"ContainerStarted","Data":"9af61727ef13126cf1a15468ad3eafe0db10833bfdb7f40dfadf4e81063a48e3"} Jul 13 18:50:13.702058 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.702034 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"65689ecefb6ab19e290de76d28675b61167d1cdf41037653ef8fd6a673a3def9"} Jul 13 18:50:13.704399 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.704377 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gxff" event={"ID":"bc5ddbf7-93cd-498b-b808-de40d5c7065b","Type":"ContainerStarted","Data":"371f9fc0b50e9fec9d139bb2b01e8ac38ca6a26ce7f6749d168b97874ad0f3dd"} Jul 13 18:50:13.708188 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.708147 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zzhs5" event={"ID":"62eaf7ee-ca92-4910-800b-e59a57f2776e","Type":"ContainerStarted","Data":"9bc2aacfbca41a267c04c8b58479a3b9086d29ff0c090198bfa956cb0e2f0ba0"} Jul 13 18:50:13.711056 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.711032 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerStarted","Data":"fa8cdcdc309c3cf2225b809a02df0c48afd201b454c3f9fad8e01282fe610f5a"} Jul 13 18:50:13.713178 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.713154 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9pjjf" event={"ID":"43347716-0baa-494f-9649-aeea5a817ac6","Type":"ContainerStarted","Data":"6892ecf686fd23a69eb5290c3f0c3d1bf59dba7a45bb0d9ad77b50b63a124f69"} Jul 13 18:50:13.716033 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:13.715984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-trm8k" event={"ID":"7b6dacaf-10a8-49ca-bfeb-1ddf56986535","Type":"ContainerStarted","Data":"ab1471840105735fa97f7e19406406232c8b8d54b3fd943786262adb3c603b20"} Jul 13 18:50:14.341607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.341573 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:14.341805 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.341638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:14.341805 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.341745 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:14.341922 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.341816 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:14.341922 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.341822 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.341801275 +0000 UTC m=+6.150507270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:14.341922 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.341895 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.34187653 +0000 UTC m=+6.150582518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:14.443458 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.442689 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:14.443458 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.442906 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:14.443458 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.442927 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:14.443458 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.442940 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:14.443458 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.443002 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.442983706 +0000 UTC m=+6.251689705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:14.667393 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.667260 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:13 +0000 UTC" deadline="2028-01-19 09:43:55.358577515 +0000 UTC" Jul 13 18:50:14.667393 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.667301 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13310h53m40.691281132s" Jul 13 18:50:14.686502 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.686459 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:14.686662 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.686604 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:14.686912 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.686471 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:14.686912 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:14.686861 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:14.687065 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.686949 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:14.687133 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:14.687054 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:16.359017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:16.358961 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:16.359492 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:16.359021 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:16.359492 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.359174 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:16.359492 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.359239 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:20.359219806 +0000 UTC m=+10.167925787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:16.359692 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.359665 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:16.359788 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.359726 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:20.359708886 +0000 UTC m=+10.168414882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:16.460149 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:16.460107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:16.460358 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.460337 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:16.460425 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.460371 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:16.460425 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.460386 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:16.460555 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.460452 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:20.460428369 +0000 UTC m=+10.269134367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:16.685621 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:16.685506 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:16.685784 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.685700 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:16.686154 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:16.686103 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:16.686244 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.686216 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:16.686312 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:16.686265 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:16.686406 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:16.686354 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:18.685484 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:18.685438 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:18.685939 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:18.685571 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:18.685939 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:18.685776 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:18.685939 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:18.685785 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:18.685939 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:18.685913 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:18.686118 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:18.685974 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:20.395068 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:20.394358 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:20.395068 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:20.394418 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:20.395068 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.394573 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:20.395068 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.394631 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:28.394613397 +0000 UTC m=+18.203319381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:20.395068 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.395000 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:20.395068 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.395058 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:28.395041825 +0000 UTC m=+18.203747804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:20.495589 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:20.495550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:20.495755 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.495739 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:20.495806 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.495766 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:20.495806 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.495779 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:20.495878 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.495851 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:28.495824242 +0000 UTC m=+18.304530234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:20.688347 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:20.688261 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:20.688347 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:20.688297 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:20.688591 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.688388 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:20.688591 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:20.688457 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:20.688591 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.688463 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:20.688591 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:20.688546 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:22.686017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:22.685985 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:22.686502 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:22.685985 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:22.686502 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:22.686105 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:22.686502 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:22.686215 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:22.686502 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:22.686314 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:22.686502 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:22.686421 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:24.685689 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:24.685654 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:24.686148 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:24.685654 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:24.686148 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:24.685780 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:24.686148 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:24.685843 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:24.686148 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:24.685949 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:24.686148 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:24.686015 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:26.685748 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:26.685540 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:26.686174 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:26.685554 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:26.686174 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:26.685838 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:26.686174 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:26.685540 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:26.686174 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:26.685935 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:26.686174 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:26.685984 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:28.450915 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:28.450879 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:28.451377 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:28.450938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:28.451377 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.451053 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:28.451377 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.451056 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:28.451377 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.451127 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:44.45110559 +0000 UTC m=+34.259811573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:28.451377 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.451147 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:44.451136751 +0000 UTC m=+34.259842736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:28.551899 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:28.551861 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:28.552056 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.552035 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:28.552056 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.552055 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:28.552126 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.552066 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:28.552126 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.552123 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:44.552104824 +0000 UTC m=+34.360810819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:28.686235 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:28.686097 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:28.686235 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:28.686115 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:28.686465 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.686250 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:28.686465 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.686327 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:28.686465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:28.686404 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:28.686653 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:28.686484 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:30.686430 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:30.686392 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:30.686868 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:30.686507 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:30.686868 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:30.686511 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:30.686868 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:30.686619 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:30.686868 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:30.686683 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:30.686868 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:30.686755 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:32.685639 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:32.685602 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:32.686153 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:32.685611 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:32.686153 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:32.685759 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:32.686153 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:32.685616 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:32.686153 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:32.685792 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:32.686153 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:32.685916 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:33.761784 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.761483 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:50:33.762539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.762180 2560 generic.go:356] "Generic (PLEG): container finished" podID="818910c4-05c0-416f-a7cc-4feee88e85c8" containerID="95db56f8608aabb5fe39e1751e01173125ac5f248234a28f5971932a62411922" exitCode=1 Jul 13 18:50:33.762539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.762266 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"bcacfec77fb73ab12d9f0f5717c2ff3b077602c6ed01fb729b7cca5e07a85c1c"} Jul 13 18:50:33.762539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.762384 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"60ec202ca10c8892b97f71a1afddfec71c11c18c317858931e859022ec96bf04"} Jul 13 18:50:33.762539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.762441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"5d224e2c38624078382c5901da03289b0f50fea9a8d62a49b8202b080de29ca1"} Jul 13 18:50:33.762539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.762489 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerDied","Data":"95db56f8608aabb5fe39e1751e01173125ac5f248234a28f5971932a62411922"} Jul 13 18:50:33.762539 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.762538 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"174c5c6341a0c23526bed43714818fd4686ebcb4a9e9acff7b55efdbf840e0ea"} Jul 13 18:50:33.763901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.763880 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zzhs5" event={"ID":"62eaf7ee-ca92-4910-800b-e59a57f2776e","Type":"ContainerStarted","Data":"fbfa1f76df648cdaca44f7e921879351b11f3e41e80e89b3e129a6a6c5911b07"} Jul 13 18:50:33.765449 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.765424 2560 generic.go:356] "Generic (PLEG): container finished" podID="774de39e-107f-49e0-9aca-8efc914d41ee" containerID="71618395aa0a43dd52522366dad32ecff825c4cdae16d0d92cf9b8689764e77f" exitCode=0 Jul 13 18:50:33.765538 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.765498 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerDied","Data":"71618395aa0a43dd52522366dad32ecff825c4cdae16d0d92cf9b8689764e77f"} Jul 13 18:50:33.767338 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.767319 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9pjjf" event={"ID":"43347716-0baa-494f-9649-aeea5a817ac6","Type":"ContainerStarted","Data":"ef7fe6f47427d4d795b4e57cfb5a6a8868853f57f518ab9117c2dbd732f5bcd8"} Jul 13 18:50:33.768660 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.768638 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-trm8k" event={"ID":"7b6dacaf-10a8-49ca-bfeb-1ddf56986535","Type":"ContainerStarted","Data":"537636aeb43663cea506454c3d60687c06ca78cbd59015e75cce772e2e34f2d3"} Jul 13 18:50:33.770010 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.769990 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" event={"ID":"eed0f0b6c98d844417e6646ccde34c95","Type":"ContainerStarted","Data":"378f59a7fa591ded3776064fb3ec844204b797c7c1c3786965c1884fd9f48dd4"} Jul 13 18:50:33.771227 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.771208 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ghf6p" event={"ID":"bcc88ca5-0a67-4ea4-a976-4d1f1a86f677","Type":"ContainerStarted","Data":"b576813cb10c826677678ae5429b6ddae7b58a7d7c152cf0ad7ca675b6bcc10f"} Jul 13 18:50:33.772647 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.772630 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" event={"ID":"89785005-2d32-4f4b-9e9d-c52feaf07769","Type":"ContainerStarted","Data":"acbffdc957e9c984681dcf146ab763e35793e2e35caa7bc4aaaeeedc35901a86"} Jul 13 18:50:33.773948 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.773929 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" event={"ID":"bac567a5-a97c-4a54-97c7-946025cc3b08","Type":"ContainerStarted","Data":"20676cecfe15fcc695f26b1068ae21df4212cd16ace2b7a2e2e782279ccef843"} Jul 13 18:50:33.775323 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.775303 2560 generic.go:356] "Generic (PLEG): container finished" podID="a0e4fc61c3258bb0daab266fcb1a4740" containerID="d59a1cdf4d20ee122ddae37b6ab49c9833795173a26cb1995f6cd9fc63e42ee5" exitCode=0 Jul 13 18:50:33.775416 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.775328 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" event={"ID":"a0e4fc61c3258bb0daab266fcb1a4740","Type":"ContainerDied","Data":"d59a1cdf4d20ee122ddae37b6ab49c9833795173a26cb1995f6cd9fc63e42ee5"} Jul 13 18:50:33.793373 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.793324 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zzhs5" podStartSLOduration=2.64882921 podStartE2EDuration="22.793305419s" totalImagesPullingTime="20.144476209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.793075791 +0000 UTC m=+23.601781791" watchObservedRunningTime="2026-07-13 18:50:33.793305419 +0000 UTC m=+23.602011421" Jul 13 18:50:33.815107 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.815073 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2hmf5" podStartSLOduration=2.668091916 podStartE2EDuration="22.815060994s" totalImagesPullingTime="20.146969078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.814914414 +0000 UTC m=+23.623620415" watchObservedRunningTime="2026-07-13 18:50:33.815060994 +0000 UTC m=+23.623766994" Jul 13 18:50:33.837048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.837012 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ghf6p" podStartSLOduration=2.602755353 podStartE2EDuration="22.837001541s" totalImagesPullingTime="20.234246188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.836342679 +0000 UTC m=+23.645048680" watchObservedRunningTime="2026-07-13 18:50:33.837001541 +0000 UTC m=+23.645707542" Jul 13 18:50:33.878542 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.878507 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9pjjf" podStartSLOduration=2.659895427 podStartE2EDuration="22.878492933s" totalImagesPullingTime="20.218597506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.878345508 +0000 UTC m=+23.687051533" watchObservedRunningTime="2026-07-13 18:50:33.878492933 +0000 UTC m=+23.687198934" Jul 13 18:50:33.922994 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:33.922913 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-27.ec2.internal" podStartSLOduration=21.922898498 podStartE2EDuration="21.922898498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.922405259 +0000 UTC m=+23.731111281" watchObservedRunningTime="2026-07-13 18:50:33.922898498 +0000 UTC m=+23.731604499" Jul 13 18:50:34.685656 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.685618 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:34.685873 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.685625 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:34.685873 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:34.685734 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:34.685873 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.685622 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:34.685873 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:34.685819 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:34.686039 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:34.685931 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:34.778976 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.778941 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" event={"ID":"a0e4fc61c3258bb0daab266fcb1a4740","Type":"ContainerStarted","Data":"186e5bc33a489f27f24a286dc62d825779dc8392dbe40b8aa2503e1dcd57861c"} Jul 13 18:50:34.781742 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.781721 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:50:34.782182 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.782160 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"9148e9312c25150f20082a2efc4ead5ee3c8d39cc1f97d4114f77d0fba2a1f75"} Jul 13 18:50:34.783501 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.783479 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gxff" event={"ID":"bc5ddbf7-93cd-498b-b808-de40d5c7065b","Type":"ContainerStarted","Data":"5225fafa169d2f41c301e1fa8bc60dbd1ed02dfef9c27f2b506d35d3da23523d"} Jul 13 18:50:34.795072 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.795038 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-trm8k" podStartSLOduration=3.584441726 podStartE2EDuration="23.795024653s" totalImagesPullingTime="20.210582927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.945922589 +0000 UTC m=+23.754628591" watchObservedRunningTime="2026-07-13 18:50:34.795024653 +0000 UTC m=+24.603730648" Jul 13 18:50:34.795390 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.795371 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-27.ec2.internal" podStartSLOduration=22.795366601 podStartE2EDuration="22.795366601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:34.795234794 +0000 UTC m=+24.603940794" watchObservedRunningTime="2026-07-13 18:50:34.795366601 +0000 UTC m=+24.604072602" Jul 13 18:50:34.811459 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:34.811423 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9gxff" podStartSLOduration=3.624468185 podStartE2EDuration="23.81141299s" totalImagesPullingTime="20.186944805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:34.811307204 +0000 UTC m=+24.620013205" watchObservedRunningTime="2026-07-13 18:50:34.81141299 +0000 UTC m=+24.620118991" Jul 13 18:50:35.142824 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:35.142802 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 18:50:35.714164 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:35.714080 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T18:50:35.142818577Z","UUID":"1abc27dc-f01b-43bc-b73c-ca1d21ae1383","Handler":null,"Name":"","Endpoint":""} Jul 13 18:50:35.715705 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:35.715683 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 18:50:35.715818 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:35.715725 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 18:50:35.790343 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:35.790206 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" event={"ID":"89785005-2d32-4f4b-9e9d-c52feaf07769","Type":"ContainerStarted","Data":"a6f6630aeababc54dc53083fe52e708b01ec406adb8b54416064057208aa8a73"} Jul 13 18:50:36.685517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.685484 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:36.685739 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.685484 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:36.685739 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.685604 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:36.685739 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:36.685603 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:36.685739 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:36.685680 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:36.685739 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:36.685736 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:36.794643 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.794605 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" event={"ID":"89785005-2d32-4f4b-9e9d-c52feaf07769","Type":"ContainerStarted","Data":"61eb6264d0d65a7293e27fa9a078d4e4ad68ff3925786baed43511d065107267"} Jul 13 18:50:36.797869 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.797849 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:50:36.798257 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.798219 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"613035c80894474f02d152c5abb52b2c6dcbb125a87160ffe144682b68c1a808"} Jul 13 18:50:36.815048 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:36.814970 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t49mr" podStartSLOduration=3.11040612 podStartE2EDuration="25.814951343s" totalImagesPullingTime="22.704545223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:36.814372018 +0000 UTC m=+26.623078017" watchObservedRunningTime="2026-07-13 18:50:36.814951343 +0000 UTC m=+26.623657543" Jul 13 18:50:38.179657 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.179481 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:38.180049 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.180029 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:38.685887 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.685449 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:38.685887 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.685455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:38.685887 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:38.685578 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:38.685887 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:38.685655 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:38.685887 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.685455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:38.685887 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:38.685751 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:38.803916 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.803515 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:38.805538 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:38.804213 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zzhs5" Jul 13 18:50:39.807212 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.807013 2560 generic.go:356] "Generic (PLEG): container finished" podID="774de39e-107f-49e0-9aca-8efc914d41ee" containerID="47b09e56a36244bb70cd85c85ed5996280f51f4856e7bce7b8f3d6022d16a641" exitCode=0 Jul 13 18:50:39.808111 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.807094 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerDied","Data":"47b09e56a36244bb70cd85c85ed5996280f51f4856e7bce7b8f3d6022d16a641"} Jul 13 18:50:39.810681 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.810665 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:50:39.811037 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.811012 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"8db83d842cca96be07c269c38c2ffa417d8a7ccf4f28494ec76d4a2fb589ee58"} Jul 13 18:50:39.811289 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.811271 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:39.811340 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.811328 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:39.811453 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.811438 2560 scope.go:117] "RemoveContainer" containerID="95db56f8608aabb5fe39e1751e01173125ac5f248234a28f5971932a62411922" Jul 13 18:50:39.826947 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:39.826929 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:40.686751 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.686719 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:40.687030 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:40.686801 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:40.687030 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.686808 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:40.687030 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.686841 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:40.687030 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:40.686876 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:40.687030 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:40.686959 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:40.816025 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.815997 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:50:40.816408 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.816353 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" event={"ID":"818910c4-05c0-416f-a7cc-4feee88e85c8","Type":"ContainerStarted","Data":"574cc9f5632f29952d5640ab5d66e1e04ec9bdf53aad5bc0b1a9176b026868ca"} Jul 13 18:50:40.816591 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.816571 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:40.833912 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.833892 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:50:40.850994 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:40.850955 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" podStartSLOduration=9.597542092 podStartE2EDuration="29.850942423s" totalImagesPullingTime="20.253400331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:40.850695141 +0000 UTC m=+30.659401141" watchObservedRunningTime="2026-07-13 18:50:40.850942423 +0000 UTC m=+30.659648424" Jul 13 18:50:41.257084 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.257042 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6mvdw"] Jul 13 18:50:41.257224 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.257178 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:41.257322 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:41.257301 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:41.259747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.259724 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6lrk"] Jul 13 18:50:41.259853 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.259808 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:41.259890 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:41.259876 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:41.262743 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.262717 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2zwjj"] Jul 13 18:50:41.262838 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.262794 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:41.262884 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:41.262865 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:41.820498 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.820468 2560 generic.go:356] "Generic (PLEG): container finished" podID="774de39e-107f-49e0-9aca-8efc914d41ee" containerID="65ff06917d72fbf22dacbf6bffc4660855af72472c3f845283c4daf075da2dd8" exitCode=0 Jul 13 18:50:41.820901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:41.820547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerDied","Data":"65ff06917d72fbf22dacbf6bffc4660855af72472c3f845283c4daf075da2dd8"} Jul 13 18:50:42.685488 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:42.685457 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:42.685669 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:42.685457 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:42.685669 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:42.685589 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:42.685669 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:42.685468 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:42.685669 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:42.685638 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:42.685873 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:42.685732 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:43.826423 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:43.826226 2560 generic.go:356] "Generic (PLEG): container finished" podID="774de39e-107f-49e0-9aca-8efc914d41ee" containerID="a083a24cd904238ef85c94b52018495940a16caea491d9d614a8817e7363db17" exitCode=0 Jul 13 18:50:43.826423 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:43.826310 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerDied","Data":"a083a24cd904238ef85c94b52018495940a16caea491d9d614a8817e7363db17"} Jul 13 18:50:44.485552 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:44.485514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:44.485743 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:44.485599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:44.485743 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.485685 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:44.485848 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.485767 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret podName:f21d64c2-eeba-4c25-92df-a13d3968ac4a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:16.485745714 +0000 UTC m=+66.294451702 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret") pod "global-pull-secret-syncer-x6lrk" (UID: "f21d64c2-eeba-4c25-92df-a13d3968ac4a") : object "kube-system"/"original-pull-secret" not registered Jul 13 18:50:44.485848 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.485690 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:44.485848 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.485842 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:51:16.485824009 +0000 UTC m=+66.294530000 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 18:50:44.586289 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:44.586253 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:44.586473 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.586388 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 18:50:44.586473 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.586403 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 18:50:44.586473 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.586413 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dk9dc for pod openshift-network-diagnostics/network-check-target-2zwjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:44.586473 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.586474 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc podName:c8e41706-6b92-4540-97b9-d2edf520c1c4 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:16.586457054 +0000 UTC m=+66.395163056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk9dc" (UniqueName: "kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc") pod "network-check-target-2zwjj" (UID: "c8e41706-6b92-4540-97b9-d2edf520c1c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 18:50:44.686054 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:44.686023 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:44.686232 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:44.686026 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:44.686232 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.686177 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6mvdw" podUID="17b4d31c-1830-4dc8-8984-9032c38bcc3e" Jul 13 18:50:44.686365 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.686254 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6lrk" podUID="f21d64c2-eeba-4c25-92df-a13d3968ac4a" Jul 13 18:50:44.686365 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:44.686026 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:44.686365 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:44.686344 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2zwjj" podUID="c8e41706-6b92-4540-97b9-d2edf520c1c4" Jul 13 18:50:45.950191 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.950114 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-27.ec2.internal" event="NodeReady" Jul 13 18:50:45.950636 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.950284 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 18:50:45.986057 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.986032 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5df6bbc64-4z85v"] Jul 13 18:50:45.990502 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.990485 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.993463 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.993434 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 18:50:45.994224 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.994201 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 18:50:45.994344 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.994275 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j66bj\"" Jul 13 18:50:45.994537 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.994520 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 18:50:45.997206 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-image-registry-private-configuration\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997329 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-trusted-ca\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997329 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997253 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-installation-pull-secrets\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997329 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997298 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vfm\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-kube-api-access-x7vfm\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997505 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997335 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-bound-sa-token\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997505 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997386 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f00938db-3c65-4995-88f3-4584c118eaab-ca-trust-extracted\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997505 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-registry-certificates\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:45.997505 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:45.997481 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.000892 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.000869 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 18:50:46.001756 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.001736 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zt6gg"] Jul 13 18:50:46.006604 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.005673 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.007858 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.007832 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5df6bbc64-4z85v"] Jul 13 18:50:46.008546 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.008527 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8l6mm"] Jul 13 18:50:46.012395 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.012367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.014637 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.014615 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 18:50:46.015174 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.015155 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 18:50:46.015348 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.015318 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rh7g\"" Jul 13 18:50:46.015517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.015383 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 18:50:46.015517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.015445 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9r5lq\"" Jul 13 18:50:46.015517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.015447 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 18:50:46.015517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.015495 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 18:50:46.028047 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.028027 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zt6gg"] Jul 13 18:50:46.037087 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.037067 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8l6mm"] Jul 13 18:50:46.098190 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-registry-certificates\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.098467 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.098467 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.098351 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:50:46.098467 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.098365 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:50:46.098467 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098417 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.098467 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.098418 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:50:46.59839928 +0000 UTC m=+36.407105273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:50:46.098467 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098467 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7657d29c-5476-4bd5-9899-ea335ae194e9-tmp-dir\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098499 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhp4\" (UniqueName: \"kubernetes.io/projected/105d01c1-59f9-416e-b67b-4a789d295814-kube-api-access-7vhp4\") pod \"ingress-canary-8l6mm\" (UID: \"105d01c1-59f9-416e-b67b-4a789d295814\") " pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098529 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105d01c1-59f9-416e-b67b-4a789d295814-cert\") pod \"ingress-canary-8l6mm\" (UID: \"105d01c1-59f9-416e-b67b-4a789d295814\") " pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098607 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-image-registry-private-configuration\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-trusted-ca\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098681 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-installation-pull-secrets\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098720 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zx5\" (UniqueName: \"kubernetes.io/projected/7657d29c-5476-4bd5-9899-ea335ae194e9-kube-api-access-n7zx5\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vfm\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-kube-api-access-x7vfm\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.098802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7657d29c-5476-4bd5-9899-ea335ae194e9-config-volume\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.099087 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-bound-sa-token\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.099087 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098845 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f00938db-3c65-4995-88f3-4584c118eaab-ca-trust-extracted\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.099087 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.098849 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-registry-certificates\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.099276 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.099231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f00938db-3c65-4995-88f3-4584c118eaab-ca-trust-extracted\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.100043 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.099827 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-trusted-ca\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.103068 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.103047 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-installation-pull-secrets\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.103068 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.103052 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-image-registry-private-configuration\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.119090 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.119045 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vfm\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-kube-api-access-x7vfm\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.119397 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.119378 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-bound-sa-token\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.199667 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.199630 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhp4\" (UniqueName: \"kubernetes.io/projected/105d01c1-59f9-416e-b67b-4a789d295814-kube-api-access-7vhp4\") pod \"ingress-canary-8l6mm\" (UID: \"105d01c1-59f9-416e-b67b-4a789d295814\") " pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.199667 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.199668 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105d01c1-59f9-416e-b67b-4a789d295814-cert\") pod \"ingress-canary-8l6mm\" (UID: \"105d01c1-59f9-416e-b67b-4a789d295814\") " pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.199888 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.199724 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zx5\" (UniqueName: \"kubernetes.io/projected/7657d29c-5476-4bd5-9899-ea335ae194e9-kube-api-access-n7zx5\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.199951 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.199889 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7657d29c-5476-4bd5-9899-ea335ae194e9-config-volume\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.200033 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.200010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.200112 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.200053 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7657d29c-5476-4bd5-9899-ea335ae194e9-tmp-dir\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.200112 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.200096 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:50:46.200254 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.200144 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:46.700130224 +0000 UTC m=+36.508836206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:50:46.200447 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.200427 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7657d29c-5476-4bd5-9899-ea335ae194e9-tmp-dir\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.200516 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.200468 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7657d29c-5476-4bd5-9899-ea335ae194e9-config-volume\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.202690 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.202666 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/105d01c1-59f9-416e-b67b-4a789d295814-cert\") pod \"ingress-canary-8l6mm\" (UID: \"105d01c1-59f9-416e-b67b-4a789d295814\") " pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.216445 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.216419 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zx5\" (UniqueName: \"kubernetes.io/projected/7657d29c-5476-4bd5-9899-ea335ae194e9-kube-api-access-n7zx5\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.219967 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.219945 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhp4\" (UniqueName: \"kubernetes.io/projected/105d01c1-59f9-416e-b67b-4a789d295814-kube-api-access-7vhp4\") pod \"ingress-canary-8l6mm\" (UID: \"105d01c1-59f9-416e-b67b-4a789d295814\") " pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.324503 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.324466 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8l6mm" Jul 13 18:50:46.603092 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.603055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:46.603284 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.603216 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:50:46.603284 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.603235 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:50:46.603406 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.603314 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:50:47.603297346 +0000 UTC m=+37.412003345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:50:46.685755 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.685721 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:50:46.685755 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.685751 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:50:46.685982 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.685721 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:50:46.688823 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.688797 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 18:50:46.688955 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.688836 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 18:50:46.688955 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.688845 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 18:50:46.689113 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.689093 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 18:50:46.690935 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.690409 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2tkcn\"" Jul 13 18:50:46.690935 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.690542 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7ddc\"" Jul 13 18:50:46.703559 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:46.703537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:46.703676 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.703657 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:50:46.703777 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:46.703718 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:47.703700429 +0000 UTC m=+37.512406424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:50:47.611588 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:47.611546 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:47.612278 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:47.611659 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:50:47.612278 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:47.611678 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:50:47.612278 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:47.611733 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:50:49.611718219 +0000 UTC m=+39.420424203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:50:47.713005 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:47.712962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:47.713167 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:47.713135 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:50:47.713258 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:47.713228 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:49.713205218 +0000 UTC m=+39.521911219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:50:47.888487 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:47.888328 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8l6mm"] Jul 13 18:50:47.892029 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:50:47.892006 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105d01c1_59f9_416e_b67b_4a789d295814.slice/crio-c7ece33d9628bfea7b488200ff6fd14b66b4eae78f5315595a5218f124256351 WatchSource:0}: Error finding container c7ece33d9628bfea7b488200ff6fd14b66b4eae78f5315595a5218f124256351: Status 404 returned error can't find the container with id c7ece33d9628bfea7b488200ff6fd14b66b4eae78f5315595a5218f124256351 Jul 13 18:50:48.843642 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:48.843601 2560 generic.go:356] "Generic (PLEG): container finished" podID="774de39e-107f-49e0-9aca-8efc914d41ee" containerID="70aa66becf037d1b39c17081b6b1e822d8263123e3aaaf9aabab78322159cb9e" exitCode=0 Jul 13 18:50:48.844184 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:48.843702 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerDied","Data":"70aa66becf037d1b39c17081b6b1e822d8263123e3aaaf9aabab78322159cb9e"} Jul 13 18:50:48.846077 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:48.846049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8l6mm" event={"ID":"105d01c1-59f9-416e-b67b-4a789d295814","Type":"ContainerStarted","Data":"c7ece33d9628bfea7b488200ff6fd14b66b4eae78f5315595a5218f124256351"} Jul 13 18:50:49.634759 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:49.634719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:49.634923 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:49.634893 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:50:49.634923 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:49.634913 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:50:49.635035 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:49.634972 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:50:53.634955031 +0000 UTC m=+43.443661012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:50:49.735656 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:49.735615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:49.735833 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:49.735795 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:50:49.735896 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:49.735862 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:53.735845969 +0000 UTC m=+43.544551967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:50:49.851479 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:49.851445 2560 generic.go:356] "Generic (PLEG): container finished" podID="774de39e-107f-49e0-9aca-8efc914d41ee" containerID="2613b378f78631dfa570d68e00b9ac914c64e33fb82813588869d31c7e741d90" exitCode=0 Jul 13 18:50:49.851900 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:49.851493 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerDied","Data":"2613b378f78631dfa570d68e00b9ac914c64e33fb82813588869d31c7e741d90"} Jul 13 18:50:50.857455 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:50.857412 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" event={"ID":"774de39e-107f-49e0-9aca-8efc914d41ee","Type":"ContainerStarted","Data":"c132b7011a47f30d6816a6595ee908b4cb5b2958257f877372a635eb694da765"} Jul 13 18:50:50.858894 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:50.858869 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8l6mm" event={"ID":"105d01c1-59f9-416e-b67b-4a789d295814","Type":"ContainerStarted","Data":"2ee2189795bf291b8d2f8489d985bb01420fbdb92ff9385bdfeb2ba210fcef94"} Jul 13 18:50:50.887574 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:50.887529 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vlhmn" podStartSLOduration=8.014872426 podStartE2EDuration="39.887513372s" totalImagesPullingTime="31.872640946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:50.885366973 +0000 UTC m=+40.694072987" watchObservedRunningTime="2026-07-13 18:50:50.887513372 +0000 UTC m=+40.696219372" Jul 13 18:50:50.902745 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:50.902662 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8l6mm" podStartSLOduration=3.146028657 podStartE2EDuration="5.902652291s" totalImagesPullingTime="2.756623634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:50.901899386 +0000 UTC m=+40.710605386" watchObservedRunningTime="2026-07-13 18:50:50.902652291 +0000 UTC m=+40.711358291" Jul 13 18:50:53.663520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:53.663477 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:50:53.663878 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:53.663628 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:50:53.663878 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:53.663648 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:50:53.663878 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:53.663701 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.663686165 +0000 UTC m=+51.472392144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:50:53.763967 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:50:53.763931 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:50:53.764103 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:53.764044 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:50:53.764103 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:50:53.764091 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.764078614 +0000 UTC m=+51.572784593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:51:01.722200 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:01.722162 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:51:01.722680 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:01.722316 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:51:01.722680 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:01.722335 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:51:01.722680 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:01.722389 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:51:17.722373752 +0000 UTC m=+67.531079731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:51:01.822898 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:01.822862 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:51:01.823030 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:01.823012 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:51:01.823085 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:01.823076 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:17.823062505 +0000 UTC m=+67.631768485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:51:12.842356 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:12.842326 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9k97" Jul 13 18:51:16.526019 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.525987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:51:16.526431 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.526043 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:51:16.528943 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.528923 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 18:51:16.528990 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.528941 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 18:51:16.536486 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:16.536468 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 18:51:16.536538 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:16.536527 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs podName:17b4d31c-1830-4dc8-8984-9032c38bcc3e nodeName:}" failed. No retries permitted until 2026-07-13 18:52:20.536510238 +0000 UTC m=+130.345216220 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs") pod "network-metrics-daemon-6mvdw" (UID: "17b4d31c-1830-4dc8-8984-9032c38bcc3e") : secret "metrics-daemon-secret" not found Jul 13 18:51:16.539838 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.539822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f21d64c2-eeba-4c25-92df-a13d3968ac4a-original-pull-secret\") pod \"global-pull-secret-syncer-x6lrk\" (UID: \"f21d64c2-eeba-4c25-92df-a13d3968ac4a\") " pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:51:16.626701 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.626665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:51:16.629787 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.629771 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 18:51:16.639656 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.639639 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 18:51:16.650132 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.650105 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9dc\" (UniqueName: \"kubernetes.io/projected/c8e41706-6b92-4540-97b9-d2edf520c1c4-kube-api-access-dk9dc\") pod \"network-check-target-2zwjj\" (UID: \"c8e41706-6b92-4540-97b9-d2edf520c1c4\") " pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:51:16.706210 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.706186 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6lrk" Jul 13 18:51:16.714683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.714664 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7ddc\"" Jul 13 18:51:16.722808 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.722789 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:51:16.829660 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.829634 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6lrk"] Jul 13 18:51:16.832608 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:51:16.832584 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21d64c2_eeba_4c25_92df_a13d3968ac4a.slice/crio-6e7391b2de641b15ae3f5668185094a967f1577588d7a6940dd3efa31ab4ea71 WatchSource:0}: Error finding container 6e7391b2de641b15ae3f5668185094a967f1577588d7a6940dd3efa31ab4ea71: Status 404 returned error can't find the container with id 6e7391b2de641b15ae3f5668185094a967f1577588d7a6940dd3efa31ab4ea71 Jul 13 18:51:16.846386 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.846306 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2zwjj"] Jul 13 18:51:16.848595 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:51:16.848572 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e41706_6b92_4540_97b9_d2edf520c1c4.slice/crio-55b5ded14c5f533ff7b9a62f555045a0bf53c815379e2f5246a2ec5e8c333294 WatchSource:0}: Error finding container 55b5ded14c5f533ff7b9a62f555045a0bf53c815379e2f5246a2ec5e8c333294: Status 404 returned error can't find the container with id 55b5ded14c5f533ff7b9a62f555045a0bf53c815379e2f5246a2ec5e8c333294 Jul 13 18:51:16.917747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.917709 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6lrk" event={"ID":"f21d64c2-eeba-4c25-92df-a13d3968ac4a","Type":"ContainerStarted","Data":"6e7391b2de641b15ae3f5668185094a967f1577588d7a6940dd3efa31ab4ea71"} Jul 13 18:51:16.918811 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:16.918789 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2zwjj" event={"ID":"c8e41706-6b92-4540-97b9-d2edf520c1c4","Type":"ContainerStarted","Data":"55b5ded14c5f533ff7b9a62f555045a0bf53c815379e2f5246a2ec5e8c333294"} Jul 13 18:51:17.735325 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:17.735284 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:51:17.735758 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:17.735452 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:51:17.735758 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:17.735467 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:51:17.735758 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:17.735525 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:51:49.735506307 +0000 UTC m=+99.544212300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:51:17.835950 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:17.835903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:51:17.836125 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:17.836076 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:51:17.836180 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:17.836135 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:49.83611683 +0000 UTC m=+99.644822809 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:51:21.935133 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:21.935035 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2zwjj" event={"ID":"c8e41706-6b92-4540-97b9-d2edf520c1c4","Type":"ContainerStarted","Data":"4c78fd671890f735c20e372722e846983eb4bd69b7e5bfd60d649f8118a7a4a1"} Jul 13 18:51:21.935595 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:21.935152 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:51:21.936368 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:21.936348 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6lrk" event={"ID":"f21d64c2-eeba-4c25-92df-a13d3968ac4a","Type":"ContainerStarted","Data":"1593bfa58da654aaccd503e227392f1a6df12db644dc32bcdc9923f505ec9438"} Jul 13 18:51:21.953274 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:21.953208 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2zwjj" podStartSLOduration=65.26470562 podStartE2EDuration="1m9.953193856s" totalImagesPullingTime="4.688488236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:21.952975825 +0000 UTC m=+71.761681820" watchObservedRunningTime="2026-07-13 18:51:21.953193856 +0000 UTC m=+71.761899835" Jul 13 18:51:31.768092 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.768035 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x6lrk" podStartSLOduration=75.052775794 podStartE2EDuration="1m19.768018211s" totalImagesPullingTime="4.715242417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:21.972205437 +0000 UTC m=+71.780911439" watchObservedRunningTime="2026-07-13 18:51:31.768018211 +0000 UTC m=+81.576724211" Jul 13 18:51:31.768548 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.768482 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-8w9l7"] Jul 13 18:51:31.771169 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.771154 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.773863 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.773842 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 13 18:51:31.774136 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.774116 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6rdzn\"" Jul 13 18:51:31.775188 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.775167 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 18:51:31.775188 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.775179 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 18:51:31.775351 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.775200 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 13 18:51:31.783276 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.783235 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 13 18:51:31.784496 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.784477 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-8w9l7"] Jul 13 18:51:31.835419 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.835392 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.835419 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.835423 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-tmp\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.835605 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.835440 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25bg2\" (UniqueName: \"kubernetes.io/projected/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-kube-api-access-25bg2\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.835605 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.835466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.835605 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.835513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-snapshots\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.835605 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.835545 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-serving-cert\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.935983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.935949 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.935983 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.935987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-tmp\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936173 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25bg2\" (UniqueName: \"kubernetes.io/projected/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-kube-api-access-25bg2\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936173 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936096 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936173 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936142 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-snapshots\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936173 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-serving-cert\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936501 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936480 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-tmp\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936601 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936582 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936776 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936756 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-snapshots\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.936892 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.936876 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.938388 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.938370 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-serving-cert\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.971466 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.971442 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-8n96k"] Jul 13 18:51:31.974427 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.974407 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:31.975442 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.975422 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25bg2\" (UniqueName: \"kubernetes.io/projected/2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2-kube-api-access-25bg2\") pod \"insights-operator-68d9dd94cf-8w9l7\" (UID: \"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2\") " pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:31.983088 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.983070 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 13 18:51:31.983413 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.983394 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 13 18:51:31.983526 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.983462 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 13 18:51:31.983588 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.983565 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4mmj5\"" Jul 13 18:51:31.986169 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.986148 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm"] Jul 13 18:51:31.988898 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.988883 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:31.998602 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.998583 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 18:51:31.998679 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:31.998644 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xgvbh\"" Jul 13 18:51:32.000152 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.000133 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 13 18:51:32.000255 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.000195 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 18:51:32.003220 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.003204 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 13 18:51:32.003592 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.003577 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 13 18:51:32.011109 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.011085 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 13 18:51:32.012694 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.012672 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm"] Jul 13 18:51:32.020082 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.020036 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-8n96k"] Jul 13 18:51:32.037555 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037532 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.037655 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037568 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5eb9e185-bf02-496f-a7c1-553e7abe225a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.037655 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9183f09e-ed22-4361-bf92-d618d4a13584-serving-cert\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.037737 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvqw\" (UniqueName: \"kubernetes.io/projected/5eb9e185-bf02-496f-a7c1-553e7abe225a-kube-api-access-mxvqw\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.037737 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037693 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9183f09e-ed22-4361-bf92-d618d4a13584-config\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.037737 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037713 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9183f09e-ed22-4361-bf92-d618d4a13584-trusted-ca\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.037737 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.037734 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskbw\" (UniqueName: \"kubernetes.io/projected/9183f09e-ed22-4361-bf92-d618d4a13584-kube-api-access-nskbw\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.079647 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.079622 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" Jul 13 18:51:32.139025 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.138955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.139025 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.139013 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5eb9e185-bf02-496f-a7c1-553e7abe225a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.139222 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.139062 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9183f09e-ed22-4361-bf92-d618d4a13584-serving-cert\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.139222 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:32.139130 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:32.139222 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:32.139207 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls podName:5eb9e185-bf02-496f-a7c1-553e7abe225a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:32.639185133 +0000 UTC m=+82.447891112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7nbqm" (UID: "5eb9e185-bf02-496f-a7c1-553e7abe225a") : secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:32.139407 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.139133 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvqw\" (UniqueName: \"kubernetes.io/projected/5eb9e185-bf02-496f-a7c1-553e7abe225a-kube-api-access-mxvqw\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.139407 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.139356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9183f09e-ed22-4361-bf92-d618d4a13584-config\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.139407 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.139397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9183f09e-ed22-4361-bf92-d618d4a13584-trusted-ca\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.139615 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.139427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskbw\" (UniqueName: \"kubernetes.io/projected/9183f09e-ed22-4361-bf92-d618d4a13584-kube-api-access-nskbw\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.140448 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.140381 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5eb9e185-bf02-496f-a7c1-553e7abe225a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.140558 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.140433 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9183f09e-ed22-4361-bf92-d618d4a13584-config\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.141169 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.141147 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9183f09e-ed22-4361-bf92-d618d4a13584-trusted-ca\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.142865 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.142837 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9183f09e-ed22-4361-bf92-d618d4a13584-serving-cert\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.150568 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.150542 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvqw\" (UniqueName: \"kubernetes.io/projected/5eb9e185-bf02-496f-a7c1-553e7abe225a-kube-api-access-mxvqw\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.151542 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.151523 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskbw\" (UniqueName: \"kubernetes.io/projected/9183f09e-ed22-4361-bf92-d618d4a13584-kube-api-access-nskbw\") pod \"console-operator-7866497494-8n96k\" (UID: \"9183f09e-ed22-4361-bf92-d618d4a13584\") " pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.203029 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.202996 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-8w9l7"] Jul 13 18:51:32.207488 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:51:32.207468 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3f7de3_fa2b_44ef_b951_f4512f9bbdb2.slice/crio-b5737c8c653aba5f5bbc0285d1bbfc708c6c5c2e2a90c72c69b560b323f695e0 WatchSource:0}: Error finding container b5737c8c653aba5f5bbc0285d1bbfc708c6c5c2e2a90c72c69b560b323f695e0: Status 404 returned error can't find the container with id b5737c8c653aba5f5bbc0285d1bbfc708c6c5c2e2a90c72c69b560b323f695e0 Jul 13 18:51:32.287926 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.287868 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:32.413197 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.413169 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-8n96k"] Jul 13 18:51:32.415930 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:51:32.415905 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9183f09e_ed22_4361_bf92_d618d4a13584.slice/crio-8ed3f67d5a5e58999f6306f9b39e2fe029c06bb19cf5c2c78d6b6c169f7bd1b9 WatchSource:0}: Error finding container 8ed3f67d5a5e58999f6306f9b39e2fe029c06bb19cf5c2c78d6b6c169f7bd1b9: Status 404 returned error can't find the container with id 8ed3f67d5a5e58999f6306f9b39e2fe029c06bb19cf5c2c78d6b6c169f7bd1b9 Jul 13 18:51:32.644541 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.644459 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:32.644685 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:32.644581 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:32.644685 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:32.644639 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls podName:5eb9e185-bf02-496f-a7c1-553e7abe225a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:33.644623729 +0000 UTC m=+83.453329714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7nbqm" (UID: "5eb9e185-bf02-496f-a7c1-553e7abe225a") : secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:32.964066 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.963984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-8n96k" event={"ID":"9183f09e-ed22-4361-bf92-d618d4a13584","Type":"ContainerStarted","Data":"8ed3f67d5a5e58999f6306f9b39e2fe029c06bb19cf5c2c78d6b6c169f7bd1b9"} Jul 13 18:51:32.965053 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:32.965033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" event={"ID":"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2","Type":"ContainerStarted","Data":"b5737c8c653aba5f5bbc0285d1bbfc708c6c5c2e2a90c72c69b560b323f695e0"} Jul 13 18:51:33.651840 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:33.651803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:33.651997 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:33.651922 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:33.651997 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:33.651972 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls podName:5eb9e185-bf02-496f-a7c1-553e7abe225a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:35.651959601 +0000 UTC m=+85.460665585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7nbqm" (UID: "5eb9e185-bf02-496f-a7c1-553e7abe225a") : secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:34.973135 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:34.973052 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" event={"ID":"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2","Type":"ContainerStarted","Data":"e49538b578aa0b986a06a92cfa7a92f9f20d0524aea7ca1987c8516548c8f510"} Jul 13 18:51:34.992795 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:34.992743 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" podStartSLOduration=1.679006707 podStartE2EDuration="3.992722903s" totalImagesPullingTime="2.313716196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:34.992236235 +0000 UTC m=+84.800942239" watchObservedRunningTime="2026-07-13 18:51:34.992722903 +0000 UTC m=+84.801428907" Jul 13 18:51:35.670009 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.669958 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:35.670189 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:35.670106 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:35.670189 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:35.670174 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls podName:5eb9e185-bf02-496f-a7c1-553e7abe225a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.670160227 +0000 UTC m=+89.478866205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7nbqm" (UID: "5eb9e185-bf02-496f-a7c1-553e7abe225a") : secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:35.961955 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.961884 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6wfw4"] Jul 13 18:51:35.964989 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.964963 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:35.968323 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.968301 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9bc6\"" Jul 13 18:51:35.968611 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.968594 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 18:51:35.969185 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.969170 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 18:51:35.980527 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.980497 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/0.log" Jul 13 18:51:35.980860 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.980549 2560 generic.go:356] "Generic (PLEG): container finished" podID="9183f09e-ed22-4361-bf92-d618d4a13584" containerID="182f04ead1e35f8586347b92b1e01ee4991d96d9bc5f7b2334de5cdd2a7e96f9" exitCode=255 Jul 13 18:51:35.980860 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.980627 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-8n96k" event={"ID":"9183f09e-ed22-4361-bf92-d618d4a13584","Type":"ContainerDied","Data":"182f04ead1e35f8586347b92b1e01ee4991d96d9bc5f7b2334de5cdd2a7e96f9"} Jul 13 18:51:35.980860 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.980789 2560 scope.go:117] "RemoveContainer" containerID="182f04ead1e35f8586347b92b1e01ee4991d96d9bc5f7b2334de5cdd2a7e96f9" Jul 13 18:51:35.987584 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:35.987527 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6wfw4"] Jul 13 18:51:36.072548 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.072496 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.072746 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.072587 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zlx\" (UniqueName: \"kubernetes.io/projected/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-api-access-p8zlx\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.072746 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.072622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.072746 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.072653 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fa2c82b-77db-4857-bbd2-61544468ab6c-data-volume\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.072939 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.072906 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fa2c82b-77db-4857-bbd2-61544468ab6c-crio-socket\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174121 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174084 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fa2c82b-77db-4857-bbd2-61544468ab6c-crio-socket\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174300 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174146 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174300 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174166 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zlx\" (UniqueName: \"kubernetes.io/projected/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-api-access-p8zlx\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174300 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174300 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fa2c82b-77db-4857-bbd2-61544468ab6c-data-volume\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174300 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174214 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fa2c82b-77db-4857-bbd2-61544468ab6c-crio-socket\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.174466 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.174310 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 18:51:36.174466 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.174322 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 18:51:36.174466 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.174385 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:36.674366357 +0000 UTC m=+86.483072350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : configmap "kube-rbac-proxy" not found Jul 13 18:51:36.174466 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.174402 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:36.674395663 +0000 UTC m=+86.483101643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : secret "insights-runtime-extractor-tls" not found Jul 13 18:51:36.174602 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.174520 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fa2c82b-77db-4857-bbd2-61544468ab6c-data-volume\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.191200 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.191177 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zlx\" (UniqueName: \"kubernetes.io/projected/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-api-access-p8zlx\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.679093 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.679059 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.679300 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.679106 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:36.679300 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.679196 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 18:51:36.679300 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.679268 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 18:51:36.679300 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.679278 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:37.67926311 +0000 UTC m=+87.487969089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : configmap "kube-rbac-proxy" not found Jul 13 18:51:36.679454 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.679317 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:37.679307033 +0000 UTC m=+87.488013011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : secret "insights-runtime-extractor-tls" not found Jul 13 18:51:36.985040 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.984972 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/1.log" Jul 13 18:51:36.985422 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.985342 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/0.log" Jul 13 18:51:36.985422 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.985375 2560 generic.go:356] "Generic (PLEG): container finished" podID="9183f09e-ed22-4361-bf92-d618d4a13584" containerID="f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e" exitCode=255 Jul 13 18:51:36.985510 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.985427 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-8n96k" event={"ID":"9183f09e-ed22-4361-bf92-d618d4a13584","Type":"ContainerDied","Data":"f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e"} Jul 13 18:51:36.985510 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.985483 2560 scope.go:117] "RemoveContainer" containerID="182f04ead1e35f8586347b92b1e01ee4991d96d9bc5f7b2334de5cdd2a7e96f9" Jul 13 18:51:36.985713 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:36.985681 2560 scope.go:117] "RemoveContainer" containerID="f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e" Jul 13 18:51:36.985876 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:36.985859 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-8n96k_openshift-console-operator(9183f09e-ed22-4361-bf92-d618d4a13584)\"" pod="openshift-console-operator/console-operator-7866497494-8n96k" podUID="9183f09e-ed22-4361-bf92-d618d4a13584" Jul 13 18:51:37.687941 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:37.687906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:37.687941 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:37.687945 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:37.688139 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:37.688029 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 18:51:37.688139 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:37.688078 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 18:51:37.688139 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:37.688089 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.688076118 +0000 UTC m=+89.496782101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : configmap "kube-rbac-proxy" not found Jul 13 18:51:37.688139 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:37.688119 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:39.688108329 +0000 UTC m=+89.496814311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : secret "insights-runtime-extractor-tls" not found Jul 13 18:51:37.749702 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:37.749679 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pjjf_43347716-0baa-494f-9649-aeea5a817ac6/dns-node-resolver/0.log" Jul 13 18:51:37.990395 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:37.990321 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/1.log" Jul 13 18:51:37.990826 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:37.990714 2560 scope.go:117] "RemoveContainer" containerID="f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e" Jul 13 18:51:37.990914 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:37.990889 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-8n96k_openshift-console-operator(9183f09e-ed22-4361-bf92-d618d4a13584)\"" pod="openshift-console-operator/console-operator-7866497494-8n96k" podUID="9183f09e-ed22-4361-bf92-d618d4a13584" Jul 13 18:51:39.149972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:39.149945 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-trm8k_7b6dacaf-10a8-49ca-bfeb-1ddf56986535/node-ca/0.log" Jul 13 18:51:39.703465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:39.703417 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:39.703465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:39.703473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:39.703681 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:39.703537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:39.703681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:39.703560 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 18:51:39.703681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:39.703634 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:39.703681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:39.703646 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.703624895 +0000 UTC m=+93.512330893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : configmap "kube-rbac-proxy" not found Jul 13 18:51:39.703681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:39.703658 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 18:51:39.703681 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:39.703673 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls podName:5eb9e185-bf02-496f-a7c1-553e7abe225a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:47.70366263 +0000 UTC m=+97.512368609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7nbqm" (UID: "5eb9e185-bf02-496f-a7c1-553e7abe225a") : secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:39.703868 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:39.703720 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.70370754 +0000 UTC m=+93.512413525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : secret "insights-runtime-extractor-tls" not found Jul 13 18:51:39.749958 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:39.749937 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l6mm_105d01c1-59f9-416e-b67b-4a789d295814/serve-healthcheck-canary/0.log" Jul 13 18:51:42.288669 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:42.288635 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:42.289038 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:42.288690 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:51:42.289038 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:42.289006 2560 scope.go:117] "RemoveContainer" containerID="f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e" Jul 13 18:51:42.289161 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:42.289142 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-8n96k_openshift-console-operator(9183f09e-ed22-4361-bf92-d618d4a13584)\"" pod="openshift-console-operator/console-operator-7866497494-8n96k" podUID="9183f09e-ed22-4361-bf92-d618d4a13584" Jul 13 18:51:43.736493 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:43.736443 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:43.736855 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:43.736504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:43.736855 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:43.736568 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 18:51:43.736855 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:43.736624 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 18:51:43.736855 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:43.736640 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:51.736625118 +0000 UTC m=+101.545331098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : configmap "kube-rbac-proxy" not found Jul 13 18:51:43.736855 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:43.736676 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:51.736657862 +0000 UTC m=+101.545363841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : secret "insights-runtime-extractor-tls" not found Jul 13 18:51:47.766955 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:47.766917 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:51:47.767441 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:47.767082 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:47.767441 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:47.767163 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls podName:5eb9e185-bf02-496f-a7c1-553e7abe225a nodeName:}" failed. No retries permitted until 2026-07-13 18:52:03.767143602 +0000 UTC m=+113.575849581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7nbqm" (UID: "5eb9e185-bf02-496f-a7c1-553e7abe225a") : secret "cluster-monitoring-operator-tls" not found Jul 13 18:51:49.784194 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:49.784154 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") pod \"image-registry-5df6bbc64-4z85v\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:51:49.784625 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:49.784321 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 18:51:49.784625 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:49.784338 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5df6bbc64-4z85v: secret "image-registry-tls" not found Jul 13 18:51:49.784625 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:49.784391 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls podName:f00938db-3c65-4995-88f3-4584c118eaab nodeName:}" failed. No retries permitted until 2026-07-13 18:52:53.784376936 +0000 UTC m=+163.593082934 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls") pod "image-registry-5df6bbc64-4z85v" (UID: "f00938db-3c65-4995-88f3-4584c118eaab") : secret "image-registry-tls" not found Jul 13 18:51:49.885341 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:49.885311 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:51:49.885496 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:49.885431 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 18:51:49.885496 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:49.885478 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls podName:7657d29c-5476-4bd5-9899-ea335ae194e9 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:53.885464745 +0000 UTC m=+163.694170723 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls") pod "dns-default-zt6gg" (UID: "7657d29c-5476-4bd5-9899-ea335ae194e9") : secret "dns-default-metrics-tls" not found Jul 13 18:51:51.803847 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:51.803812 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:51.803847 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:51.803851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:51.804411 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:51.803982 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 18:51:51.804411 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:51.804036 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls podName:7fa2c82b-77db-4857-bbd2-61544468ab6c nodeName:}" failed. No retries permitted until 2026-07-13 18:52:07.80402202 +0000 UTC m=+117.612728002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wfw4" (UID: "7fa2c82b-77db-4857-bbd2-61544468ab6c") : secret "insights-runtime-extractor-tls" not found Jul 13 18:51:51.804541 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:51.804513 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fa2c82b-77db-4857-bbd2-61544468ab6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:51:52.943115 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:52.943090 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2zwjj" Jul 13 18:51:57.685928 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:57.685899 2560 scope.go:117] "RemoveContainer" containerID="f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e" Jul 13 18:51:58.047194 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:58.047166 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 18:51:58.047518 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:58.047503 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/1.log" Jul 13 18:51:58.047566 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:58.047538 2560 generic.go:356] "Generic (PLEG): container finished" podID="9183f09e-ed22-4361-bf92-d618d4a13584" containerID="5bd259459bb2435df675b65851a1a5f846089ea07d7fde1edcf60f72514ac4fc" exitCode=255 Jul 13 18:51:58.047601 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:58.047584 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-8n96k" event={"ID":"9183f09e-ed22-4361-bf92-d618d4a13584","Type":"ContainerDied","Data":"5bd259459bb2435df675b65851a1a5f846089ea07d7fde1edcf60f72514ac4fc"} Jul 13 18:51:58.047649 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:58.047637 2560 scope.go:117] "RemoveContainer" containerID="f38ed839a0c54b6eac2904ebbe06bf99c52ee9113c2bc0e8e0462e78b6d65a9e" Jul 13 18:51:58.047956 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:58.047938 2560 scope.go:117] "RemoveContainer" containerID="5bd259459bb2435df675b65851a1a5f846089ea07d7fde1edcf60f72514ac4fc" Jul 13 18:51:58.048127 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:51:58.048105 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-8n96k_openshift-console-operator(9183f09e-ed22-4361-bf92-d618d4a13584)\"" pod="openshift-console-operator/console-operator-7866497494-8n96k" podUID="9183f09e-ed22-4361-bf92-d618d4a13584" Jul 13 18:51:59.052070 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:51:59.052040 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 18:52:01.790544 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.790516 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t"] Jul 13 18:52:01.793323 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.793308 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:01.797577 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.797556 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 18:52:01.798200 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.798171 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 18:52:01.799012 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.798994 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 18:52:01.799103 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.799013 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 13 18:52:01.809570 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.809545 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t"] Jul 13 18:52:01.986278 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.986224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxdc\" (UniqueName: \"kubernetes.io/projected/9e355469-1208-4fde-8527-1712cb8280d8-kube-api-access-ksxdc\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:01.986457 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.986412 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9e355469-1208-4fde-8527-1712cb8280d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:01.986520 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:01.986473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e355469-1208-4fde-8527-1712cb8280d8-tmp\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.086857 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.086786 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9e355469-1208-4fde-8527-1712cb8280d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.086857 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.086824 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e355469-1208-4fde-8527-1712cb8280d8-tmp\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.087064 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.086957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxdc\" (UniqueName: \"kubernetes.io/projected/9e355469-1208-4fde-8527-1712cb8280d8-kube-api-access-ksxdc\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.087166 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.087150 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e355469-1208-4fde-8527-1712cb8280d8-tmp\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.089227 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.089201 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9e355469-1208-4fde-8527-1712cb8280d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.116924 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.116896 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxdc\" (UniqueName: \"kubernetes.io/projected/9e355469-1208-4fde-8527-1712cb8280d8-kube-api-access-ksxdc\") pod \"klusterlet-addon-workmgr-869d47bb48-h5k6t\" (UID: \"9e355469-1208-4fde-8527-1712cb8280d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.288432 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.288401 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:52:02.288602 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.288453 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:52:02.288797 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.288784 2560 scope.go:117] "RemoveContainer" containerID="5bd259459bb2435df675b65851a1a5f846089ea07d7fde1edcf60f72514ac4fc" Jul 13 18:52:02.288954 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:02.288934 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-8n96k_openshift-console-operator(9183f09e-ed22-4361-bf92-d618d4a13584)\"" pod="openshift-console-operator/console-operator-7866497494-8n96k" podUID="9183f09e-ed22-4361-bf92-d618d4a13584" Jul 13 18:52:02.403042 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.402957 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:02.522130 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:02.522108 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t"] Jul 13 18:52:02.524295 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:02.524268 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e355469_1208_4fde_8527_1712cb8280d8.slice/crio-6590ed17ff54f32a4ad5fb57c778da5afa9fb29a05dacd0dee72a1c1654575f2 WatchSource:0}: Error finding container 6590ed17ff54f32a4ad5fb57c778da5afa9fb29a05dacd0dee72a1c1654575f2: Status 404 returned error can't find the container with id 6590ed17ff54f32a4ad5fb57c778da5afa9fb29a05dacd0dee72a1c1654575f2 Jul 13 18:52:03.065921 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:03.065856 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" event={"ID":"9e355469-1208-4fde-8527-1712cb8280d8","Type":"ContainerStarted","Data":"6590ed17ff54f32a4ad5fb57c778da5afa9fb29a05dacd0dee72a1c1654575f2"} Jul 13 18:52:03.800866 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:03.800821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:52:03.803405 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:03.803379 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5eb9e185-bf02-496f-a7c1-553e7abe225a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7nbqm\" (UID: \"5eb9e185-bf02-496f-a7c1-553e7abe225a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:52:04.096973 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:04.096887 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" Jul 13 18:52:04.236137 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:04.236103 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm"] Jul 13 18:52:04.239638 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:04.239612 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb9e185_bf02_496f_a7c1_553e7abe225a.slice/crio-b1d4b1e59baeabcd87c4722dcafa469916e0bc6f81b1b3cbad04174d245ede92 WatchSource:0}: Error finding container b1d4b1e59baeabcd87c4722dcafa469916e0bc6f81b1b3cbad04174d245ede92: Status 404 returned error can't find the container with id b1d4b1e59baeabcd87c4722dcafa469916e0bc6f81b1b3cbad04174d245ede92 Jul 13 18:52:05.074526 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:05.074455 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" event={"ID":"5eb9e185-bf02-496f-a7c1-553e7abe225a","Type":"ContainerStarted","Data":"b1d4b1e59baeabcd87c4722dcafa469916e0bc6f81b1b3cbad04174d245ede92"} Jul 13 18:52:07.828022 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:07.827983 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:52:07.830547 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:07.830520 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fa2c82b-77db-4857-bbd2-61544468ab6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wfw4\" (UID: \"7fa2c82b-77db-4857-bbd2-61544468ab6c\") " pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:52:08.073391 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.073371 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6wfw4" Jul 13 18:52:08.265377 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.265231 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6wfw4"] Jul 13 18:52:08.275366 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:08.275339 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa2c82b_77db_4857_bbd2_61544468ab6c.slice/crio-19f70d7bc0bb6642f21aab8e001662eb807c3499c30734f8800793f66a020508 WatchSource:0}: Error finding container 19f70d7bc0bb6642f21aab8e001662eb807c3499c30734f8800793f66a020508: Status 404 returned error can't find the container with id 19f70d7bc0bb6642f21aab8e001662eb807c3499c30734f8800793f66a020508 Jul 13 18:52:08.576982 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.576951 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls"] Jul 13 18:52:08.579690 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.579670 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:08.582180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.582159 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 13 18:52:08.582725 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.582702 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tf8ds\"" Jul 13 18:52:08.590440 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.590421 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls"] Jul 13 18:52:08.634496 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.634468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pwls\" (UID: \"a31c45ef-7f94-473b-ac36-f42e2e8c2509\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:08.735486 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:08.735451 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pwls\" (UID: \"a31c45ef-7f94-473b-ac36-f42e2e8c2509\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:08.735624 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:08.735583 2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 13 18:52:08.735664 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:08.735642 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates podName:a31c45ef-7f94-473b-ac36-f42e2e8c2509 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:09.23562756 +0000 UTC m=+119.044333538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-6pwls" (UID: "a31c45ef-7f94-473b-ac36-f42e2e8c2509") : secret "prometheus-operator-admission-webhook-tls" not found Jul 13 18:52:09.092719 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.092681 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" event={"ID":"9e355469-1208-4fde-8527-1712cb8280d8","Type":"ContainerStarted","Data":"d71709581d44de21d305a321c62d0f6e853cb6889a2b7b75a9441d0a5f7f1920"} Jul 13 18:52:09.093170 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.092825 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:09.094910 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.094883 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" event={"ID":"5eb9e185-bf02-496f-a7c1-553e7abe225a","Type":"ContainerStarted","Data":"b28e5692bce4d162f8ed9576503df7fa981e136cb3b54a23851c19df57ec54be"} Jul 13 18:52:09.095458 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.095439 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" Jul 13 18:52:09.097727 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.097670 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wfw4" event={"ID":"7fa2c82b-77db-4857-bbd2-61544468ab6c","Type":"ContainerStarted","Data":"19f70d7bc0bb6642f21aab8e001662eb807c3499c30734f8800793f66a020508"} Jul 13 18:52:09.114898 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.114392 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869d47bb48-h5k6t" podStartSLOduration=2.637912456 podStartE2EDuration="8.114375227s" totalImagesPullingTime="5.476462771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:09.113363933 +0000 UTC m=+118.922069937" watchObservedRunningTime="2026-07-13 18:52:09.114375227 +0000 UTC m=+118.923081228" Jul 13 18:52:09.135470 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.135291 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" podStartSLOduration=34.370698183 podStartE2EDuration="38.13527539s" totalImagesPullingTime="3.764577207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:09.134261112 +0000 UTC m=+118.942967108" watchObservedRunningTime="2026-07-13 18:52:09.13527539 +0000 UTC m=+118.943981387" Jul 13 18:52:09.241358 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:09.241188 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pwls\" (UID: \"a31c45ef-7f94-473b-ac36-f42e2e8c2509\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:09.241358 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:09.241352 2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 13 18:52:09.241600 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:09.241420 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates podName:a31c45ef-7f94-473b-ac36-f42e2e8c2509 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:10.241401006 +0000 UTC m=+120.050106985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-6pwls" (UID: "a31c45ef-7f94-473b-ac36-f42e2e8c2509") : secret "prometheus-operator-admission-webhook-tls" not found Jul 13 18:52:10.247359 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:10.247317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pwls\" (UID: \"a31c45ef-7f94-473b-ac36-f42e2e8c2509\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:10.249911 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:10.249889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a31c45ef-7f94-473b-ac36-f42e2e8c2509-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6pwls\" (UID: \"a31c45ef-7f94-473b-ac36-f42e2e8c2509\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:10.388722 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:10.388687 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:12.043914 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:12.043786 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls"] Jul 13 18:52:12.046221 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:12.046200 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31c45ef_7f94_473b_ac36_f42e2e8c2509.slice/crio-8c589879ae3ff57ae21eade70b38111c7add8fb8d1493eef266ce0cca7c156bb WatchSource:0}: Error finding container 8c589879ae3ff57ae21eade70b38111c7add8fb8d1493eef266ce0cca7c156bb: Status 404 returned error can't find the container with id 8c589879ae3ff57ae21eade70b38111c7add8fb8d1493eef266ce0cca7c156bb Jul 13 18:52:12.107517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:12.107483 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" event={"ID":"a31c45ef-7f94-473b-ac36-f42e2e8c2509","Type":"ContainerStarted","Data":"8c589879ae3ff57ae21eade70b38111c7add8fb8d1493eef266ce0cca7c156bb"} Jul 13 18:52:12.108872 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:12.108851 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wfw4" event={"ID":"7fa2c82b-77db-4857-bbd2-61544468ab6c","Type":"ContainerStarted","Data":"7a24471f2e25f78227cfbf8dcebfbea910b3541054b957627c08b5a4e8e8a76a"} Jul 13 18:52:14.116500 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:14.116465 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" event={"ID":"a31c45ef-7f94-473b-ac36-f42e2e8c2509","Type":"ContainerStarted","Data":"669bff7bfd1c842b0b9d9b8905b19e53e4185a1a1fb81497ac9dc8626bd6701f"} Jul 13 18:52:14.116993 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:14.116605 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:14.118275 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:14.118251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wfw4" event={"ID":"7fa2c82b-77db-4857-bbd2-61544468ab6c","Type":"ContainerStarted","Data":"b6b1c1432cff19afa6899115349babf480e50707bd25484159a93831a7aea9ba"} Jul 13 18:52:14.121121 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:14.121105 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" Jul 13 18:52:14.134281 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:14.134223 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6pwls" podStartSLOduration=4.972456018 podStartE2EDuration="6.134212791s" totalImagesPullingTime="1.161756773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:14.13334106 +0000 UTC m=+123.942047073" watchObservedRunningTime="2026-07-13 18:52:14.134212791 +0000 UTC m=+123.942918769" Jul 13 18:52:15.686031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:15.685994 2560 scope.go:117] "RemoveContainer" containerID="5bd259459bb2435df675b65851a1a5f846089ea07d7fde1edcf60f72514ac4fc" Jul 13 18:52:15.686452 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:15.686171 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-8n96k_openshift-console-operator(9183f09e-ed22-4361-bf92-d618d4a13584)\"" pod="openshift-console-operator/console-operator-7866497494-8n96k" podUID="9183f09e-ed22-4361-bf92-d618d4a13584" Jul 13 18:52:18.136031 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:18.135997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wfw4" event={"ID":"7fa2c82b-77db-4857-bbd2-61544468ab6c","Type":"ContainerStarted","Data":"fb4f7fbffdc45fe1d4461a197b69bdd01c1e6aa30a431e4a7db91c0819bebaf1"} Jul 13 18:52:18.155332 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:18.155292 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6wfw4" podStartSLOduration=33.865345304 podStartE2EDuration="43.155277217s" totalImagesPullingTime="9.289931913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:51:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:18.154642141 +0000 UTC m=+127.963348152" watchObservedRunningTime="2026-07-13 18:52:18.155277217 +0000 UTC m=+127.963983222" Jul 13 18:52:20.626200 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:20.626157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:52:20.628411 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:20.628383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b4d31c-1830-4dc8-8984-9032c38bcc3e-metrics-certs\") pod \"network-metrics-daemon-6mvdw\" (UID: \"17b4d31c-1830-4dc8-8984-9032c38bcc3e\") " pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:52:20.902364 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:20.902286 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2tkcn\"" Jul 13 18:52:20.909210 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:20.909189 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6mvdw" Jul 13 18:52:21.034783 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:21.034761 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6mvdw"] Jul 13 18:52:21.036730 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:21.036695 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b4d31c_1830_4dc8_8984_9032c38bcc3e.slice/crio-73dd233fc725387edc06de40478af86bdaaf393bd90104cd7527f027f766df11 WatchSource:0}: Error finding container 73dd233fc725387edc06de40478af86bdaaf393bd90104cd7527f027f766df11: Status 404 returned error can't find the container with id 73dd233fc725387edc06de40478af86bdaaf393bd90104cd7527f027f766df11 Jul 13 18:52:21.146768 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:21.146734 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6mvdw" event={"ID":"17b4d31c-1830-4dc8-8984-9032c38bcc3e","Type":"ContainerStarted","Data":"73dd233fc725387edc06de40478af86bdaaf393bd90104cd7527f027f766df11"} Jul 13 18:52:23.156759 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:23.156681 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 18:52:23.156759 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:23.156723 2560 generic.go:356] "Generic (PLEG): container finished" podID="5eb9e185-bf02-496f-a7c1-553e7abe225a" containerID="b28e5692bce4d162f8ed9576503df7fa981e136cb3b54a23851c19df57ec54be" exitCode=2 Jul 13 18:52:23.157284 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:23.156800 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" event={"ID":"5eb9e185-bf02-496f-a7c1-553e7abe225a","Type":"ContainerDied","Data":"b28e5692bce4d162f8ed9576503df7fa981e136cb3b54a23851c19df57ec54be"} Jul 13 18:52:23.157284 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:23.157109 2560 scope.go:117] "RemoveContainer" containerID="b28e5692bce4d162f8ed9576503df7fa981e136cb3b54a23851c19df57ec54be" Jul 13 18:52:23.811205 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:23.811174 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5df6bbc64-4z85v"] Jul 13 18:52:23.811480 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:23.811459 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" podUID="f00938db-3c65-4995-88f3-4584c118eaab" Jul 13 18:52:24.161557 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.161474 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 18:52:24.161937 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.161592 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7nbqm" event={"ID":"5eb9e185-bf02-496f-a7c1-553e7abe225a","Type":"ContainerStarted","Data":"b7c2509d509a29a4f755e01de2af93438485546d05b93aa170e759be5004eb14"} Jul 13 18:52:24.163270 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.163252 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:52:24.163270 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.163266 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6mvdw" event={"ID":"17b4d31c-1830-4dc8-8984-9032c38bcc3e","Type":"ContainerStarted","Data":"ae5af9c57847e34836b5bb8c9f28db4eb3b2414abcc39823b13fc01aa73867aa"} Jul 13 18:52:24.163416 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.163302 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6mvdw" event={"ID":"17b4d31c-1830-4dc8-8984-9032c38bcc3e","Type":"ContainerStarted","Data":"7b5f3a890f8c4f29afcc6ecf5a46d694e990d1f5d1b51db665353efefb80eb08"} Jul 13 18:52:24.169403 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.169386 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:52:24.229941 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.229887 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6mvdw" podStartSLOduration=131.012438778 podStartE2EDuration="2m13.229869579s" totalImagesPullingTime="2.217430801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:24.228577559 +0000 UTC m=+134.037283560" watchObservedRunningTime="2026-07-13 18:52:24.229869579 +0000 UTC m=+134.038575581" Jul 13 18:52:24.252839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.252820 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-trusted-ca\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.252922 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.252854 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-bound-sa-token\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.252922 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.252880 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-image-registry-private-configuration\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.252922 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.252898 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f00938db-3c65-4995-88f3-4584c118eaab-ca-trust-extracted\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.252922 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.252916 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-registry-certificates\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.253115 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.252991 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7vfm\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-kube-api-access-x7vfm\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.253115 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.253024 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-installation-pull-secrets\") pod \"f00938db-3c65-4995-88f3-4584c118eaab\" (UID: \"f00938db-3c65-4995-88f3-4584c118eaab\") " Jul 13 18:52:24.253220 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.253176 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00938db-3c65-4995-88f3-4584c118eaab-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 18:52:24.253312 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.253289 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:52:24.253371 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.253320 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f00938db-3c65-4995-88f3-4584c118eaab-ca-trust-extracted\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:24.253371 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.253319 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:52:24.255294 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.255268 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:52:24.255392 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.255338 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:52:24.255392 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.255369 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:52:24.255662 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.255645 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-kube-api-access-x7vfm" (OuterVolumeSpecName: "kube-api-access-x7vfm") pod "f00938db-3c65-4995-88f3-4584c118eaab" (UID: "f00938db-3c65-4995-88f3-4584c118eaab"). InnerVolumeSpecName "kube-api-access-x7vfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:52:24.353682 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.353638 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vfm\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-kube-api-access-x7vfm\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:24.353682 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.353677 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-installation-pull-secrets\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:24.353682 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.353686 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-trusted-ca\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:24.353682 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.353695 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-bound-sa-token\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:24.353682 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.353704 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f00938db-3c65-4995-88f3-4584c118eaab-image-registry-private-configuration\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:24.353965 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:24.353713 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f00938db-3c65-4995-88f3-4584c118eaab-registry-certificates\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:25.166349 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:25.166319 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5df6bbc64-4z85v" Jul 13 18:52:25.217005 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:25.216975 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5df6bbc64-4z85v"] Jul 13 18:52:25.219767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:25.219746 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5df6bbc64-4z85v"] Jul 13 18:52:25.260967 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:25.260942 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f00938db-3c65-4995-88f3-4584c118eaab-registry-tls\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:52:26.689009 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.688976 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00938db-3c65-4995-88f3-4584c118eaab" path="/var/lib/kubelet/pods/f00938db-3c65-4995-88f3-4584c118eaab/volumes" Jul 13 18:52:26.794058 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.794025 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn"] Jul 13 18:52:26.798193 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.798178 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.800997 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.800975 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 13 18:52:26.801151 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.801016 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 13 18:52:26.802499 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.802484 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 18:52:26.802499 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.802492 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xg4z4\"" Jul 13 18:52:26.808329 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.808310 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn"] Jul 13 18:52:26.874644 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.874611 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97602007-12cc-41fe-8d23-e6f91a2c4d42-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.874784 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.874676 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97602007-12cc-41fe-8d23-e6f91a2c4d42-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.874784 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.874712 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/97602007-12cc-41fe-8d23-e6f91a2c4d42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.874784 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.874753 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62rq\" (UniqueName: \"kubernetes.io/projected/97602007-12cc-41fe-8d23-e6f91a2c4d42-kube-api-access-s62rq\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.938534 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.938500 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-4ldfb"] Jul 13 18:52:26.941642 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.941600 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.945518 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.945496 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 13 18:52:26.945692 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.945668 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ccxp8\"" Jul 13 18:52:26.946104 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.946083 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 13 18:52:26.946205 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.946144 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 13 18:52:26.961500 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.961472 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-scltw"] Jul 13 18:52:26.964426 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.964408 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-4ldfb"] Jul 13 18:52:26.964537 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.964508 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:26.967770 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.967752 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6pnlk\"" Jul 13 18:52:26.967988 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.967972 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 18:52:26.968081 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.967975 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 18:52:26.968081 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.968016 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 18:52:26.975508 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975488 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.975583 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975520 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.975583 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975547 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97602007-12cc-41fe-8d23-e6f91a2c4d42-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.975583 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kc7\" (UniqueName: \"kubernetes.io/projected/17a3af64-6172-479a-bf5c-2a50b624546a-kube-api-access-78kc7\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.975691 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975616 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/97602007-12cc-41fe-8d23-e6f91a2c4d42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.975691 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975650 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.975691 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62rq\" (UniqueName: \"kubernetes.io/projected/97602007-12cc-41fe-8d23-e6f91a2c4d42-kube-api-access-s62rq\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.975784 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975701 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a3af64-6172-479a-bf5c-2a50b624546a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.975784 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17a3af64-6172-479a-bf5c-2a50b624546a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:26.975846 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.975787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97602007-12cc-41fe-8d23-e6f91a2c4d42-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.976176 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.976146 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97602007-12cc-41fe-8d23-e6f91a2c4d42-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.978072 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.978047 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97602007-12cc-41fe-8d23-e6f91a2c4d42-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.978166 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.978147 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/97602007-12cc-41fe-8d23-e6f91a2c4d42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:26.991632 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:26.991605 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62rq\" (UniqueName: \"kubernetes.io/projected/97602007-12cc-41fe-8d23-e6f91a2c4d42-kube-api-access-s62rq\") pod \"openshift-state-metrics-84bd68b896-6zkxn\" (UID: \"97602007-12cc-41fe-8d23-e6f91a2c4d42\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:27.077081 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077044 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-root\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077081 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g42n\" (UniqueName: \"kubernetes.io/projected/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-kube-api-access-4g42n\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077142 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077163 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-metrics-client-ca\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077204 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-textfile\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077312 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kc7\" (UniqueName: \"kubernetes.io/projected/17a3af64-6172-479a-bf5c-2a50b624546a-kube-api-access-78kc7\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.077384 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.077710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077402 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-sys\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077426 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a3af64-6172-479a-bf5c-2a50b624546a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.077710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17a3af64-6172-479a-bf5c-2a50b624546a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.077710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077485 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-tls\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-wtmp\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.077929 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077789 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17a3af64-6172-479a-bf5c-2a50b624546a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.078010 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.077987 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.078069 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.078034 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a3af64-6172-479a-bf5c-2a50b624546a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.079562 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.079540 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.079643 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.079634 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17a3af64-6172-479a-bf5c-2a50b624546a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.088329 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.088311 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kc7\" (UniqueName: \"kubernetes.io/projected/17a3af64-6172-479a-bf5c-2a50b624546a-kube-api-access-78kc7\") pod \"kube-state-metrics-785778ff74-4ldfb\" (UID: \"17a3af64-6172-479a-bf5c-2a50b624546a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.107150 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.107126 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" Jul 13 18:52:27.178325 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178296 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178464 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-textfile\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178464 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178386 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-sys\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178464 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178417 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-tls\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178464 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178440 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-wtmp\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178674 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-root\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178674 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178467 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-sys\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178674 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178491 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g42n\" (UniqueName: \"kubernetes.io/projected/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-kube-api-access-4g42n\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178674 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178541 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178674 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-metrics-client-ca\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178674 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178625 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-wtmp\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178951 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-root\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.178951 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.178751 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-textfile\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.179143 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.179124 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-metrics-client-ca\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.179269 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.179223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.180732 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.180707 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-tls\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.180883 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.180862 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.191314 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.191274 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g42n\" (UniqueName: \"kubernetes.io/projected/56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a-kube-api-access-4g42n\") pod \"node-exporter-scltw\" (UID: \"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a\") " pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.247111 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.247083 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn"] Jul 13 18:52:27.249719 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:27.249697 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97602007_12cc_41fe_8d23_e6f91a2c4d42.slice/crio-d96960969b550fcef0d171f393e71ec7c9fe612c742f2c3ec95bf62375aebb00 WatchSource:0}: Error finding container d96960969b550fcef0d171f393e71ec7c9fe612c742f2c3ec95bf62375aebb00: Status 404 returned error can't find the container with id d96960969b550fcef0d171f393e71ec7c9fe612c742f2c3ec95bf62375aebb00 Jul 13 18:52:27.250548 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.250530 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" Jul 13 18:52:27.273620 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.273596 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-scltw" Jul 13 18:52:27.286846 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:27.286821 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c2bb7b_dc60_476a_bbe6_f8c369f5ed2a.slice/crio-3c9ac18e05b2b548a6f4a8b076474646f2bae5389778996d8a8103ef598f9a70 WatchSource:0}: Error finding container 3c9ac18e05b2b548a6f4a8b076474646f2bae5389778996d8a8103ef598f9a70: Status 404 returned error can't find the container with id 3c9ac18e05b2b548a6f4a8b076474646f2bae5389778996d8a8103ef598f9a70 Jul 13 18:52:27.397090 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:27.397065 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-4ldfb"] Jul 13 18:52:27.399446 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:27.399417 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a3af64_6172_479a_bf5c_2a50b624546a.slice/crio-109f1634d93484e0541a1b16d44b0660c5af7dfb79506f49309529c338d59953 WatchSource:0}: Error finding container 109f1634d93484e0541a1b16d44b0660c5af7dfb79506f49309529c338d59953: Status 404 returned error can't find the container with id 109f1634d93484e0541a1b16d44b0660c5af7dfb79506f49309529c338d59953 Jul 13 18:52:28.179102 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:28.179066 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" event={"ID":"97602007-12cc-41fe-8d23-e6f91a2c4d42","Type":"ContainerStarted","Data":"f776ab81641f7b49d30f7f9b8e4ec25535396a49aece7790a0b6c8e1861ce1e5"} Jul 13 18:52:28.179549 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:28.179136 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" event={"ID":"97602007-12cc-41fe-8d23-e6f91a2c4d42","Type":"ContainerStarted","Data":"45fce413aaa1929313b30c19a9312a141e3410f232dc2eda77591e88fe978b3c"} Jul 13 18:52:28.179549 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:28.179166 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" event={"ID":"97602007-12cc-41fe-8d23-e6f91a2c4d42","Type":"ContainerStarted","Data":"d96960969b550fcef0d171f393e71ec7c9fe612c742f2c3ec95bf62375aebb00"} Jul 13 18:52:28.180471 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:28.180441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scltw" event={"ID":"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a","Type":"ContainerStarted","Data":"3c9ac18e05b2b548a6f4a8b076474646f2bae5389778996d8a8103ef598f9a70"} Jul 13 18:52:28.181873 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:28.181848 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" event={"ID":"17a3af64-6172-479a-bf5c-2a50b624546a","Type":"ContainerStarted","Data":"109f1634d93484e0541a1b16d44b0660c5af7dfb79506f49309529c338d59953"} Jul 13 18:52:28.686124 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:28.686093 2560 scope.go:117] "RemoveContainer" containerID="5bd259459bb2435df675b65851a1a5f846089ea07d7fde1edcf60f72514ac4fc" Jul 13 18:52:29.188325 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.188295 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 18:52:29.188737 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.188454 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-8n96k" event={"ID":"9183f09e-ed22-4361-bf92-d618d4a13584","Type":"ContainerStarted","Data":"e351603574d2cd6177912f6a4e253534828b061eeeecafc643b5ca0efce3b728"} Jul 13 18:52:29.189307 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.189221 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:52:29.192196 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.192163 2560 generic.go:356] "Generic (PLEG): container finished" podID="56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a" containerID="4553eadc28e47651846f82eba5786bef56658eae3a27f3acefbee5e1bba00435" exitCode=0 Jul 13 18:52:29.192318 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.192282 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scltw" event={"ID":"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a","Type":"ContainerDied","Data":"4553eadc28e47651846f82eba5786bef56658eae3a27f3acefbee5e1bba00435"} Jul 13 18:52:29.198705 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.198652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" event={"ID":"17a3af64-6172-479a-bf5c-2a50b624546a","Type":"ContainerStarted","Data":"5d0681338263c3e1793bc572f046866b78ef36662ee7f3b0c3ce5067c97cb37e"} Jul 13 18:52:29.198705 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.198706 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" event={"ID":"17a3af64-6172-479a-bf5c-2a50b624546a","Type":"ContainerStarted","Data":"06cc7212b90bc4566b03a73ba13e7bb73ca2548748b077c9892ac4b069464bae"} Jul 13 18:52:29.213960 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.213912 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-8n96k" podStartSLOduration=55.424849239 podStartE2EDuration="58.213893287s" totalImagesPullingTime="2.789044048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:29.212631004 +0000 UTC m=+139.021337007" watchObservedRunningTime="2026-07-13 18:52:29.213893287 +0000 UTC m=+139.022599291" Jul 13 18:52:29.780938 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.780910 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-8n96k" Jul 13 18:52:29.981914 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.981879 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-hs6ww"] Jul 13 18:52:29.985091 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.985071 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:29.988271 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.988230 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 18:52:29.988400 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.988378 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 18:52:29.988469 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.988446 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lckkc\"" Jul 13 18:52:29.998896 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:29.998875 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-hs6ww"] Jul 13 18:52:30.100595 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.100491 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qqg\" (UniqueName: \"kubernetes.io/projected/13c22c75-d2bc-4ab9-9a15-10d91df7f2d7-kube-api-access-n6qqg\") pod \"downloads-68c9cbb497-hs6ww\" (UID: \"13c22c75-d2bc-4ab9-9a15-10d91df7f2d7\") " pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:30.201216 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.201183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qqg\" (UniqueName: \"kubernetes.io/projected/13c22c75-d2bc-4ab9-9a15-10d91df7f2d7-kube-api-access-n6qqg\") pod \"downloads-68c9cbb497-hs6ww\" (UID: \"13c22c75-d2bc-4ab9-9a15-10d91df7f2d7\") " pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:30.204368 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.204318 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" event={"ID":"97602007-12cc-41fe-8d23-e6f91a2c4d42","Type":"ContainerStarted","Data":"8c7771f0b923bb42feb01b970e9928f63425a6360f86eb4c9e63487014bb06fb"} Jul 13 18:52:30.206734 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.206706 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scltw" event={"ID":"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a","Type":"ContainerStarted","Data":"641a500a72033453ae7a4673a12833dcaeab82db5fa556b4e96da1c067beb26a"} Jul 13 18:52:30.206833 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.206754 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scltw" event={"ID":"56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a","Type":"ContainerStarted","Data":"0532ded1d3fe64ba2d79dce35ebffe179406fd687dd29db25cb3bf30273adbfb"} Jul 13 18:52:30.208738 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.208718 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" event={"ID":"17a3af64-6172-479a-bf5c-2a50b624546a","Type":"ContainerStarted","Data":"0201e1c5a124e1cc3256a7a9334ae73e224458f2db5cc64a345f5b18ca6a29c0"} Jul 13 18:52:30.213999 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.213978 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qqg\" (UniqueName: \"kubernetes.io/projected/13c22c75-d2bc-4ab9-9a15-10d91df7f2d7-kube-api-access-n6qqg\") pod \"downloads-68c9cbb497-hs6ww\" (UID: \"13c22c75-d2bc-4ab9-9a15-10d91df7f2d7\") " pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:30.227990 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.227952 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6zkxn" podStartSLOduration=1.7707873969999999 podStartE2EDuration="4.227940835s" totalImagesPullingTime="2.457153438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:30.226329008 +0000 UTC m=+140.035035005" watchObservedRunningTime="2026-07-13 18:52:30.227940835 +0000 UTC m=+140.036646886" Jul 13 18:52:30.252843 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.252783 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-4ldfb" podStartSLOduration=2.656296296 podStartE2EDuration="4.252766556s" totalImagesPullingTime="1.59647026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:30.25079452 +0000 UTC m=+140.059500533" watchObservedRunningTime="2026-07-13 18:52:30.252766556 +0000 UTC m=+140.061472559" Jul 13 18:52:30.279861 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.279814 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-scltw" podStartSLOduration=3.046910591 podStartE2EDuration="4.279800006s" totalImagesPullingTime="1.157630231s" totalInitContainerRuntime="75.259184ms" isStatefulPod=true podCreationTimestamp="2026-07-13 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:30.279134513 +0000 UTC m=+140.087840540" watchObservedRunningTime="2026-07-13 18:52:30.279800006 +0000 UTC m=+140.088506040" Jul 13 18:52:30.293857 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.293838 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:30.420158 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.420128 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-hs6ww"] Jul 13 18:52:30.423134 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:30.423105 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c22c75_d2bc_4ab9_9a15_10d91df7f2d7.slice/crio-4cf69b0a84a34ce626161bad4b5cc6b642f6be573dee89600eb547dfdfad72e5 WatchSource:0}: Error finding container 4cf69b0a84a34ce626161bad4b5cc6b642f6be573dee89600eb547dfdfad72e5: Status 404 returned error can't find the container with id 4cf69b0a84a34ce626161bad4b5cc6b642f6be573dee89600eb547dfdfad72e5 Jul 13 18:52:30.941419 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.941385 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-696c94d74d-fczz9"] Jul 13 18:52:30.946974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.946388 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:30.949646 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949620 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 13 18:52:30.949767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949687 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7kcdfiiosrcfc\"" Jul 13 18:52:30.949767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949704 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 13 18:52:30.949767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949661 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 13 18:52:30.949767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949663 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-c7kwd\"" Jul 13 18:52:30.949767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949685 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 13 18:52:30.949974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.949653 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 13 18:52:30.963180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:30.963155 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-696c94d74d-fczz9"] Jul 13 18:52:31.008719 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4g2\" (UniqueName: \"kubernetes.io/projected/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-kube-api-access-hn4g2\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.008851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008733 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.008851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008755 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.008851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008834 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-tls\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.008952 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.008952 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008926 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-metrics-client-ca\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.009022 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.008952 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-grpc-tls\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.009022 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.009010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109452 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109404 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-grpc-tls\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109627 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109627 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4g2\" (UniqueName: \"kubernetes.io/projected/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-kube-api-access-hn4g2\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109627 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109568 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109627 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109829 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-tls\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109829 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109667 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.109829 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.109704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-metrics-client-ca\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.110462 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.110433 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-metrics-client-ca\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.112624 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.112597 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-tls\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.112729 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.112633 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-grpc-tls\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.112729 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.112636 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.113013 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.112990 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.113320 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.113300 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.113684 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.113658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.121747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.121722 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4g2\" (UniqueName: \"kubernetes.io/projected/5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7-kube-api-access-hn4g2\") pod \"thanos-querier-696c94d74d-fczz9\" (UID: \"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7\") " pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.214703 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.214600 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-hs6ww" event={"ID":"13c22c75-d2bc-4ab9-9a15-10d91df7f2d7","Type":"ContainerStarted","Data":"4cf69b0a84a34ce626161bad4b5cc6b642f6be573dee89600eb547dfdfad72e5"} Jul 13 18:52:31.220222 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.220198 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6646549cdd-m8z9p"] Jul 13 18:52:31.224698 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.224681 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.227505 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.227483 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 13 18:52:31.227636 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.227535 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 13 18:52:31.227636 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.227539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7wmtw\"" Jul 13 18:52:31.227772 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.227756 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 13 18:52:31.227826 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.227756 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1dn9dr1c3tpct\"" Jul 13 18:52:31.227942 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.227927 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 18:52:31.234664 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.234644 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6646549cdd-m8z9p"] Jul 13 18:52:31.256337 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.256316 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:31.311492 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311463 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.311592 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311511 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtn4q\" (UniqueName: \"kubernetes.io/projected/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-kube-api-access-gtn4q\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.311659 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311603 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-secret-metrics-server-client-certs\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.312747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-metrics-server-audit-profiles\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.312747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311765 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-secret-metrics-server-tls\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.312747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311806 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-audit-log\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.312747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.311835 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-client-ca-bundle\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.404258 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.404132 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-696c94d74d-fczz9"] Jul 13 18:52:31.407105 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:31.407067 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceb26f4_f49c_4a92_8b9e_cdabf6d093a7.slice/crio-b22cb8cd4bc37b1fceef84bb617c62888c4427ffaac8bd99b5d81bea621161f8 WatchSource:0}: Error finding container b22cb8cd4bc37b1fceef84bb617c62888c4427ffaac8bd99b5d81bea621161f8: Status 404 returned error can't find the container with id b22cb8cd4bc37b1fceef84bb617c62888c4427ffaac8bd99b5d81bea621161f8 Jul 13 18:52:31.414133 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414028 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.414133 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414069 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtn4q\" (UniqueName: \"kubernetes.io/projected/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-kube-api-access-gtn4q\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.414133 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414104 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-secret-metrics-server-client-certs\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.414362 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414142 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-metrics-server-audit-profiles\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.414362 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414168 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-secret-metrics-server-tls\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.414362 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414313 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-audit-log\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.414362 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.414358 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-client-ca-bundle\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.415193 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.415167 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-metrics-server-audit-profiles\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.415193 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.415183 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-audit-log\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.415607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.415585 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.417065 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.417025 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-secret-metrics-server-client-certs\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.417387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.417366 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-secret-metrics-server-tls\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.417506 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.417486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-client-ca-bundle\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.425404 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.425382 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtn4q\" (UniqueName: \"kubernetes.io/projected/710dc3cc-7e3f-4bf3-94b9-79c1ca18b517-kube-api-access-gtn4q\") pod \"metrics-server-6646549cdd-m8z9p\" (UID: \"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517\") " pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.537026 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.536994 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:31.708716 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.708682 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6646549cdd-m8z9p"] Jul 13 18:52:31.711640 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:31.711615 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710dc3cc_7e3f_4bf3_94b9_79c1ca18b517.slice/crio-38b1fdf06a58167d8a671d622043165a398d1229c1c9ab8ee9c1d486f16a6f08 WatchSource:0}: Error finding container 38b1fdf06a58167d8a671d622043165a398d1229c1c9ab8ee9c1d486f16a6f08: Status 404 returned error can't find the container with id 38b1fdf06a58167d8a671d622043165a398d1229c1c9ab8ee9c1d486f16a6f08 Jul 13 18:52:31.795758 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.795668 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg"] Jul 13 18:52:31.798842 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.798815 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:31.802849 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.802823 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 13 18:52:31.802984 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.802870 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jtt6g\"" Jul 13 18:52:31.812668 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.812617 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg"] Jul 13 18:52:31.918920 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.918888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/406cc36e-9b15-4f1f-ae67-be9cd23f3007-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t2zjg\" (UID: \"406cc36e-9b15-4f1f-ae67-be9cd23f3007\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:31.919082 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:31.918925 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66wx\" (UniqueName: \"kubernetes.io/projected/406cc36e-9b15-4f1f-ae67-be9cd23f3007-kube-api-access-z66wx\") pod \"monitoring-plugin-5f68df7f58-t2zjg\" (UID: \"406cc36e-9b15-4f1f-ae67-be9cd23f3007\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:32.020361 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.020314 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/406cc36e-9b15-4f1f-ae67-be9cd23f3007-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t2zjg\" (UID: \"406cc36e-9b15-4f1f-ae67-be9cd23f3007\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:32.020544 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.020370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66wx\" (UniqueName: \"kubernetes.io/projected/406cc36e-9b15-4f1f-ae67-be9cd23f3007-kube-api-access-z66wx\") pod \"monitoring-plugin-5f68df7f58-t2zjg\" (UID: \"406cc36e-9b15-4f1f-ae67-be9cd23f3007\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:32.022839 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.022817 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/406cc36e-9b15-4f1f-ae67-be9cd23f3007-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t2zjg\" (UID: \"406cc36e-9b15-4f1f-ae67-be9cd23f3007\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:32.029165 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.029134 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66wx\" (UniqueName: \"kubernetes.io/projected/406cc36e-9b15-4f1f-ae67-be9cd23f3007-kube-api-access-z66wx\") pod \"monitoring-plugin-5f68df7f58-t2zjg\" (UID: \"406cc36e-9b15-4f1f-ae67-be9cd23f3007\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:32.110448 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.110365 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:32.139055 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.139025 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c"] Jul 13 18:52:32.151899 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.151860 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.158317 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.157618 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 13 18:52:32.158317 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.157618 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 13 18:52:32.158317 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.158055 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 13 18:52:32.158317 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.158063 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 13 18:52:32.158597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.158323 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4m7gq\"" Jul 13 18:52:32.158651 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.158601 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 13 18:52:32.166095 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.166076 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 13 18:52:32.176597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.176570 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c"] Jul 13 18:52:32.221997 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.221888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-secret-telemeter-client\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.221997 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.221954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-telemeter-client-tls\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.222921 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.222823 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-serving-certs-ca-bundle\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.222921 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.222873 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mxg\" (UniqueName: \"kubernetes.io/projected/ddbad9e6-fcf2-40b6-8f84-e63e81450972-kube-api-access-h5mxg\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.223073 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.223005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-metrics-client-ca\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.223132 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.223072 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.223132 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.223108 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-federate-client-tls\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.223224 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.223134 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.228413 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.228116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" event={"ID":"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517","Type":"ContainerStarted","Data":"38b1fdf06a58167d8a671d622043165a398d1229c1c9ab8ee9c1d486f16a6f08"} Jul 13 18:52:32.230827 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.230753 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"b22cb8cd4bc37b1fceef84bb617c62888c4427ffaac8bd99b5d81bea621161f8"} Jul 13 18:52:32.286968 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.286941 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg"] Jul 13 18:52:32.289418 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:32.289387 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406cc36e_9b15_4f1f_ae67_be9cd23f3007.slice/crio-eaee1bb8d2c1505db122fc76baf3b3010b6524cdf0e837751fff9147000626b5 WatchSource:0}: Error finding container eaee1bb8d2c1505db122fc76baf3b3010b6524cdf0e837751fff9147000626b5: Status 404 returned error can't find the container with id eaee1bb8d2c1505db122fc76baf3b3010b6524cdf0e837751fff9147000626b5 Jul 13 18:52:32.323925 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.323890 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-federate-client-tls\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324079 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.323946 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324079 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.323979 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-secret-telemeter-client\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324079 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.324017 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-telemeter-client-tls\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324079 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.324056 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-serving-certs-ca-bundle\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324807 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.324082 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mxg\" (UniqueName: \"kubernetes.io/projected/ddbad9e6-fcf2-40b6-8f84-e63e81450972-kube-api-access-h5mxg\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324807 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.324155 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-metrics-client-ca\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.324807 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.324180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.325069 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.325002 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-serving-certs-ca-bundle\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.325176 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.325150 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-metrics-client-ca\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.325230 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.325202 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbad9e6-fcf2-40b6-8f84-e63e81450972-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.327562 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.327469 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-secret-telemeter-client\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.327562 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.327528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.327562 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.327545 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-telemeter-client-tls\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.327562 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.327490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddbad9e6-fcf2-40b6-8f84-e63e81450972-federate-client-tls\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.334458 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.334434 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mxg\" (UniqueName: \"kubernetes.io/projected/ddbad9e6-fcf2-40b6-8f84-e63e81450972-kube-api-access-h5mxg\") pod \"telemeter-client-5f8d48d6bb-l4z5c\" (UID: \"ddbad9e6-fcf2-40b6-8f84-e63e81450972\") " pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.466437 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.466360 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" Jul 13 18:52:32.646502 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:32.646480 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c"] Jul 13 18:52:32.648152 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:32.648129 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbad9e6_fcf2_40b6_8f84_e63e81450972.slice/crio-d7f5d2e70bd49ecee12b03a751a62d7e6cb62d98d398c5d7ec9afd0d346efaaf WatchSource:0}: Error finding container d7f5d2e70bd49ecee12b03a751a62d7e6cb62d98d398c5d7ec9afd0d346efaaf: Status 404 returned error can't find the container with id d7f5d2e70bd49ecee12b03a751a62d7e6cb62d98d398c5d7ec9afd0d346efaaf Jul 13 18:52:33.164802 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.164770 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:52:33.168814 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.168784 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.174333 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.174311 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 18:52:33.176560 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.176325 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 18:52:33.176560 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.176389 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 18:52:33.176560 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.176496 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 18:52:33.176779 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.176673 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 18:52:33.176779 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.176735 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 18:52:33.176882 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.176793 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fh87d\"" Jul 13 18:52:33.177483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.177455 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 18:52:33.177612 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.177596 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8gatmmofm2q8q\"" Jul 13 18:52:33.177681 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.177594 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 18:52:33.177681 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.177674 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 18:52:33.178114 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.178055 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 18:52:33.178502 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.178318 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 18:52:33.180789 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.180767 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 18:52:33.181935 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.181912 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 18:52:33.182738 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.182714 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 18:52:33.196085 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.196053 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:52:33.232453 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232453 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232458 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232478 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232542 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232594 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-config\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232714 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232737 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232843 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.232940 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232914 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.233525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.232979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.233525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.233040 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hh9\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-kube-api-access-b6hh9\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.233525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.233084 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.233525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.233112 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.233525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.233144 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.233525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.233183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.236737 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.236708 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" event={"ID":"ddbad9e6-fcf2-40b6-8f84-e63e81450972","Type":"ContainerStarted","Data":"d7f5d2e70bd49ecee12b03a751a62d7e6cb62d98d398c5d7ec9afd0d346efaaf"} Jul 13 18:52:33.238180 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.238143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" event={"ID":"406cc36e-9b15-4f1f-ae67-be9cd23f3007","Type":"ContainerStarted","Data":"eaee1bb8d2c1505db122fc76baf3b3010b6524cdf0e837751fff9147000626b5"} Jul 13 18:52:33.334414 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334414 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334114 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334414 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334288 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334414 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334332 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334414 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hh9\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-kube-api-access-b6hh9\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334414 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334472 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334514 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334720 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334761 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334826 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334855 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334910 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.334974 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334951 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-config\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.335334 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.334978 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.335334 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.335008 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.335334 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.335058 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.335334 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.335189 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.339302 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.338736 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.339302 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.339074 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.339498 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.339353 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.340170 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.340140 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.340415 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.340387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.341119 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.341059 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.341225 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.341125 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.341529 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.341480 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.344655 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.344637 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.345022 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.344992 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.345480 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.345430 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-config\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.346055 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.345950 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.346528 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.346218 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.346528 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.346381 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.346528 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.346480 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.348024 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.347989 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hh9\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-kube-api-access-b6hh9\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.348143 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.348123 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.348729 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.348711 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.482371 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.482299 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:33.973659 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:33.973324 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:52:33.974492 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:33.974466 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caa78aa_cb18_4101_a954_663154527a0b.slice/crio-c16a35d39294058300bcee1a4d4f73dc48de1f0dc7c8b59bd2321949c70f1456 WatchSource:0}: Error finding container c16a35d39294058300bcee1a4d4f73dc48de1f0dc7c8b59bd2321949c70f1456: Status 404 returned error can't find the container with id c16a35d39294058300bcee1a4d4f73dc48de1f0dc7c8b59bd2321949c70f1456 Jul 13 18:52:34.249691 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:34.249650 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" event={"ID":"406cc36e-9b15-4f1f-ae67-be9cd23f3007","Type":"ContainerStarted","Data":"61bbbcf6dd17e4291a1cbd674240da5d3d8a10264ffec0aaa799ece54e67e169"} Jul 13 18:52:34.250603 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:34.250523 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:34.252510 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:34.251970 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"c16a35d39294058300bcee1a4d4f73dc48de1f0dc7c8b59bd2321949c70f1456"} Jul 13 18:52:34.257099 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:34.257053 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" Jul 13 18:52:34.274171 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:34.274122 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t2zjg" podStartSLOduration=1.737482037 podStartE2EDuration="3.274103004s" totalImagesPullingTime="1.536620967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:34.272060934 +0000 UTC m=+144.080766935" watchObservedRunningTime="2026-07-13 18:52:34.274103004 +0000 UTC m=+144.082808990" Jul 13 18:52:37.268505 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.268471 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"16869b7eb1fcbc72d8a87685aa24ecbcc55f007f12ab85b774c9161cf99bdb20"} Jul 13 18:52:37.268962 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.268563 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"db791431a992e30f41632c1e11aab08194bf7ba0bca145ba40f129d863fa8cfd"} Jul 13 18:52:37.268962 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.268618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"a4a2d32add09a7877d3791df34d2a560162c530cb55f694e702b13cc1fcf7e86"} Jul 13 18:52:37.272922 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.272890 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" exitCode=0 Jul 13 18:52:37.273039 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.273000 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} Jul 13 18:52:37.275596 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.275547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" event={"ID":"710dc3cc-7e3f-4bf3-94b9-79c1ca18b517","Type":"ContainerStarted","Data":"92cce2d6a9c20d9c1f6dd1c833f7af53e1d3b5f36935e217e14881b9c0c565de"} Jul 13 18:52:37.277981 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.277948 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" event={"ID":"ddbad9e6-fcf2-40b6-8f84-e63e81450972","Type":"ContainerStarted","Data":"7a385aa71fe2c8f42e4d40d1852281ba06626897368106d0c1daf2e06867c17e"} Jul 13 18:52:37.278068 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.278005 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" event={"ID":"ddbad9e6-fcf2-40b6-8f84-e63e81450972","Type":"ContainerStarted","Data":"a5b8d5491862c35ca613fec22c94f6f9efa6d19ca6704483b16c3260b0aa6f27"} Jul 13 18:52:37.278068 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.278037 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" event={"ID":"ddbad9e6-fcf2-40b6-8f84-e63e81450972","Type":"ContainerStarted","Data":"a542006a16d3d2039f02471348594c92fe12e3e1caf22ef9205c05e1f0c6ece1"} Jul 13 18:52:37.327493 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.327437 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" podStartSLOduration=1.469490597 podStartE2EDuration="6.327418789s" totalImagesPullingTime="4.857928192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:37.32608675 +0000 UTC m=+147.134792752" watchObservedRunningTime="2026-07-13 18:52:37.327418789 +0000 UTC m=+147.136124791" Jul 13 18:52:37.356047 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:37.355971 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f8d48d6bb-l4z5c" podStartSLOduration=1.4375013939999999 podStartE2EDuration="5.355953293s" totalImagesPullingTime="3.918451899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:37.353114099 +0000 UTC m=+147.161820118" watchObservedRunningTime="2026-07-13 18:52:37.355953293 +0000 UTC m=+147.164659294" Jul 13 18:52:39.291985 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:39.291953 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"0e658a5ac883cd48df57b7feffa5b4c55d54715f12c429977a60337eb36d86fa"} Jul 13 18:52:39.292441 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:39.292023 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"e6fb09cf5fe073e5cd214930d445ec274e0e8f5160b9d2cc4bef1bba17264155"} Jul 13 18:52:39.292441 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:39.292058 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" event={"ID":"5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7","Type":"ContainerStarted","Data":"b807523140fbd6460ef5939a3a2d1465bd51a5b56fca71b29f63555d4f47830b"} Jul 13 18:52:39.292441 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:39.292191 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:39.318282 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:39.318218 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" podStartSLOduration=2.315871925 podStartE2EDuration="9.318199806s" totalImagesPullingTime="7.002327881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:39.317006745 +0000 UTC m=+149.125712797" watchObservedRunningTime="2026-07-13 18:52:39.318199806 +0000 UTC m=+149.126905808" Jul 13 18:52:41.308632 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:41.308204 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} Jul 13 18:52:41.308632 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:41.308304 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} Jul 13 18:52:42.317740 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:42.317699 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} Jul 13 18:52:42.318163 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:42.317780 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} Jul 13 18:52:42.318163 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:42.317821 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} Jul 13 18:52:42.318163 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:42.317861 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerStarted","Data":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} Jul 13 18:52:42.350354 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:42.350294 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.049620345 podStartE2EDuration="9.350277115s" totalImagesPullingTime="6.276018378s" totalInitContainerRuntime="24.638392ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:42.348707385 +0000 UTC m=+152.157413387" watchObservedRunningTime="2026-07-13 18:52:42.350277115 +0000 UTC m=+152.158983117" Jul 13 18:52:43.482664 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:43.482628 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:52:45.304775 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:45.304749 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-696c94d74d-fczz9" Jul 13 18:52:49.017999 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:52:49.017951 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zt6gg" podUID="7657d29c-5476-4bd5-9899-ea335ae194e9" Jul 13 18:52:49.353750 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:49.353659 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-hs6ww" event={"ID":"13c22c75-d2bc-4ab9-9a15-10d91df7f2d7","Type":"ContainerStarted","Data":"396c5353add0ca9c83cf7168478cddace3cd8194f1a2a0fcb0aeaa2dc425ddb1"} Jul 13 18:52:49.353916 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:49.353745 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt6gg" Jul 13 18:52:49.354334 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:49.354028 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:49.374149 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:49.374122 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-hs6ww" Jul 13 18:52:49.376976 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:49.376938 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-hs6ww" podStartSLOduration=1.752993947 podStartE2EDuration="20.37692207s" totalImagesPullingTime="18.623928123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:49.374485688 +0000 UTC m=+159.183191689" watchObservedRunningTime="2026-07-13 18:52:49.37692207 +0000 UTC m=+159.185628132" Jul 13 18:52:51.537662 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:51.537623 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:51.538182 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:51.537699 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:52:53.954946 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:53.954905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:52:53.957567 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:53.957538 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7657d29c-5476-4bd5-9899-ea335ae194e9-metrics-tls\") pod \"dns-default-zt6gg\" (UID: \"7657d29c-5476-4bd5-9899-ea335ae194e9\") " pod="openshift-dns/dns-default-zt6gg" Jul 13 18:52:54.157692 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:54.157661 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rh7g\"" Jul 13 18:52:54.165496 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:54.165471 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt6gg" Jul 13 18:52:54.338525 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:54.338492 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zt6gg"] Jul 13 18:52:54.342604 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:52:54.342566 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7657d29c_5476_4bd5_9899_ea335ae194e9.slice/crio-26a358cbf281d51784287b63d81141de83f34b434d0c2814c88c04366b4f8831 WatchSource:0}: Error finding container 26a358cbf281d51784287b63d81141de83f34b434d0c2814c88c04366b4f8831: Status 404 returned error can't find the container with id 26a358cbf281d51784287b63d81141de83f34b434d0c2814c88c04366b4f8831 Jul 13 18:52:54.384840 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:54.384802 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt6gg" event={"ID":"7657d29c-5476-4bd5-9899-ea335ae194e9","Type":"ContainerStarted","Data":"26a358cbf281d51784287b63d81141de83f34b434d0c2814c88c04366b4f8831"} Jul 13 18:52:55.393787 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:55.393749 2560 generic.go:356] "Generic (PLEG): container finished" podID="2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2" containerID="e49538b578aa0b986a06a92cfa7a92f9f20d0524aea7ca1987c8516548c8f510" exitCode=0 Jul 13 18:52:55.394228 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:55.393823 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" event={"ID":"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2","Type":"ContainerDied","Data":"e49538b578aa0b986a06a92cfa7a92f9f20d0524aea7ca1987c8516548c8f510"} Jul 13 18:52:55.394367 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:55.394348 2560 scope.go:117] "RemoveContainer" containerID="e49538b578aa0b986a06a92cfa7a92f9f20d0524aea7ca1987c8516548c8f510" Jul 13 18:52:56.402447 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:56.402409 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8w9l7" event={"ID":"2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2","Type":"ContainerStarted","Data":"6484c1ec3ea104d1dabec309a581feda21bf7a98876ca453f42123311e3c2f2f"} Jul 13 18:52:57.409629 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:57.409582 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt6gg" event={"ID":"7657d29c-5476-4bd5-9899-ea335ae194e9","Type":"ContainerStarted","Data":"c4f7b717bb2c752acf0804db855e5b161453107432330432eef2339fa9f8879a"} Jul 13 18:52:58.415110 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:58.415076 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt6gg" event={"ID":"7657d29c-5476-4bd5-9899-ea335ae194e9","Type":"ContainerStarted","Data":"2e690d4dd9fe0d17d6809be10d0b6011403bcb12c4e4caa1bb03b012c50ade1e"} Jul 13 18:52:58.415634 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:58.415206 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zt6gg" Jul 13 18:52:58.440990 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:58.437481 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zt6gg" podStartSLOduration=130.642469537 podStartE2EDuration="2m13.437458865s" totalImagesPullingTime="2.794989328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:58.434396337 +0000 UTC m=+168.243102339" watchObservedRunningTime="2026-07-13 18:52:58.437458865 +0000 UTC m=+168.246164866" Jul 13 18:52:58.644427 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:58.644400 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 18:52:58.867138 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:58.867108 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 18:52:59.043878 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:59.043847 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/1.log" Jul 13 18:52:59.243001 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:59.242932 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4ldfb_17a3af64-6172-479a-bf5c-2a50b624546a/kube-state-metrics/0.log" Jul 13 18:52:59.443804 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:59.443779 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4ldfb_17a3af64-6172-479a-bf5c-2a50b624546a/kube-rbac-proxy-main/0.log" Jul 13 18:52:59.642043 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:59.642018 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4ldfb_17a3af64-6172-479a-bf5c-2a50b624546a/kube-rbac-proxy-self/0.log" Jul 13 18:52:59.842878 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:52:59.842855 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6646549cdd-m8z9p_710dc3cc-7e3f-4bf3-94b9-79c1ca18b517/metrics-server/0.log" Jul 13 18:53:00.042334 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:00.042308 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-t2zjg_406cc36e-9b15-4f1f-ae67-be9cd23f3007/monitoring-plugin/0.log" Jul 13 18:53:01.443092 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:01.443063 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scltw_56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a/init-textfile/0.log" Jul 13 18:53:01.643858 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:01.643832 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scltw_56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a/node-exporter/0.log" Jul 13 18:53:01.842822 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:01.842800 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scltw_56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a/kube-rbac-proxy/0.log" Jul 13 18:53:02.042926 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:02.042893 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6zkxn_97602007-12cc-41fe-8d23-e6f91a2c4d42/kube-rbac-proxy-main/0.log" Jul 13 18:53:02.244278 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:02.244188 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6zkxn_97602007-12cc-41fe-8d23-e6f91a2c4d42/kube-rbac-proxy-self/0.log" Jul 13 18:53:02.442377 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:02.442347 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6zkxn_97602007-12cc-41fe-8d23-e6f91a2c4d42/openshift-state-metrics/0.log" Jul 13 18:53:02.643091 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:02.643066 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/init-config-reloader/0.log" Jul 13 18:53:02.844078 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:02.844052 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/prometheus/0.log" Jul 13 18:53:03.043777 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:03.043740 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/config-reloader/0.log" Jul 13 18:53:03.243288 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:03.243176 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/thanos-sidecar/0.log" Jul 13 18:53:03.443391 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:03.443313 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/kube-rbac-proxy-web/0.log" Jul 13 18:53:03.643615 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:03.643588 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/kube-rbac-proxy/0.log" Jul 13 18:53:03.843252 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:03.843213 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4caa78aa-cb18-4101-a954-663154527a0b/kube-rbac-proxy-thanos/0.log" Jul 13 18:53:04.443005 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:04.442971 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-6pwls_a31c45ef-7f94-473b-ac36-f42e2e8c2509/prometheus-operator-admission-webhook/0.log" Jul 13 18:53:04.644875 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:04.644839 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8d48d6bb-l4z5c_ddbad9e6-fcf2-40b6-8f84-e63e81450972/telemeter-client/0.log" Jul 13 18:53:04.842751 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:04.842714 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8d48d6bb-l4z5c_ddbad9e6-fcf2-40b6-8f84-e63e81450972/reload/0.log" Jul 13 18:53:05.049788 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:05.049755 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8d48d6bb-l4z5c_ddbad9e6-fcf2-40b6-8f84-e63e81450972/kube-rbac-proxy/0.log" Jul 13 18:53:05.243838 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:05.243740 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/thanos-query/0.log" Jul 13 18:53:05.448640 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:05.448611 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy-web/0.log" Jul 13 18:53:05.643093 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:05.643043 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy/0.log" Jul 13 18:53:05.843378 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:05.843347 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/prom-label-proxy/0.log" Jul 13 18:53:06.043003 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:06.042975 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy-rules/0.log" Jul 13 18:53:06.243201 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:06.243158 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy-metrics/0.log" Jul 13 18:53:06.643005 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:06.642978 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 18:53:06.844591 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:06.844562 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/3.log" Jul 13 18:53:07.447397 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:07.447371 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-hs6ww_13c22c75-d2bc-4ab9-9a15-10d91df7f2d7/download-server/0.log" Jul 13 18:53:08.042887 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:08.042853 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l6mm_105d01c1-59f9-416e-b67b-4a789d295814/serve-healthcheck-canary/0.log" Jul 13 18:53:08.421209 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:08.421133 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zt6gg" Jul 13 18:53:11.543483 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:11.543453 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:53:11.548162 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:11.548137 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6646549cdd-m8z9p" Jul 13 18:53:33.483462 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:33.483416 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:33.552846 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:33.552818 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:33.571938 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:33.571912 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:51.522772 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.522732 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:53:51.523268 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.523013 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="prometheus" containerID="cri-o://32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" gracePeriod=600 Jul 13 18:53:51.523268 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.523053 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="thanos-sidecar" containerID="cri-o://c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" gracePeriod=600 Jul 13 18:53:51.523268 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.523057 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy" containerID="cri-o://03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" gracePeriod=600 Jul 13 18:53:51.523268 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.523100 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="config-reloader" containerID="cri-o://535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" gracePeriod=600 Jul 13 18:53:51.523268 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.523113 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-web" containerID="cri-o://5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" gracePeriod=600 Jul 13 18:53:51.523268 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.523113 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" gracePeriod=600 Jul 13 18:53:51.764859 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.764832 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:51.858508 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858433 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-2\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858508 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858497 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-trusted-ca-bundle\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858723 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858599 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-tls-assets\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858723 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858635 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-config\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858723 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858656 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-0\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858723 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858686 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858723 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858714 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-metrics-client-ca\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858739 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-metrics-client-certs\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858755 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858759 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858790 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-kube-rbac-proxy\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858820 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-grpc-tls\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858848 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858858 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-kubelet-serving-ca-bundle\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.858972 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.858916 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-config-out\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859061 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-serving-certs-ca-bundle\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859092 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-db\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859137 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-thanos-prometheus-http-client-file\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859164 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6hh9\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-kube-api-access-b6hh9\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859173 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859190 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-web-config\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859215 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-1\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859387 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859276 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-tls\") pod \"4caa78aa-cb18-4101-a954-663154527a0b\" (UID: \"4caa78aa-cb18-4101-a954-663154527a0b\") " Jul 13 18:53:51.859760 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859542 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.859760 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859560 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.859760 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.859575 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.860211 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.860172 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.860989 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.860959 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.861691 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.861296 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.861691 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.861658 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:53:51.862006 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.861977 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:53:51.862259 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862176 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.862354 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862298 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.862424 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862382 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.862424 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862403 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-config" (OuterVolumeSpecName: "config") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.862620 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862596 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 18:53:51.862739 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.862980 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862955 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.863045 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.862982 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.863362 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.863332 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-config-out" (OuterVolumeSpecName: "config-out") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 18:53:51.863523 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.863507 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-kube-api-access-b6hh9" (OuterVolumeSpecName: "kube-api-access-b6hh9") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "kube-api-access-b6hh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:53:51.864178 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.864161 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.874407 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.874386 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-web-config" (OuterVolumeSpecName: "web-config") pod "4caa78aa-cb18-4101-a954-663154527a0b" (UID: "4caa78aa-cb18-4101-a954-663154527a0b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:53:51.960889 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960868 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-config-out\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.960889 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960888 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960896 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-db\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960907 2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960916 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hh9\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-kube-api-access-b6hh9\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960924 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-web-config\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960932 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960940 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960949 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4caa78aa-cb18-4101-a954-663154527a0b-tls-assets\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960958 2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-config\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960966 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960974 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960983 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4caa78aa-cb18-4101-a954-663154527a0b-configmap-metrics-client-ca\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.960991 2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-metrics-client-certs\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.961000 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.961007 2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-kube-rbac-proxy\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:51.961011 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:51.961015 2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4caa78aa-cb18-4101-a954-663154527a0b-secret-grpc-tls\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 18:53:52.642206 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642167 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" exitCode=0 Jul 13 18:53:52.642206 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642201 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" exitCode=0 Jul 13 18:53:52.642206 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642211 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" exitCode=0 Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642219 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" exitCode=0 Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642227 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" exitCode=0 Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642235 2560 generic.go:356] "Generic (PLEG): container finished" podID="4caa78aa-cb18-4101-a954-663154527a0b" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" exitCode=0 Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642263 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642285 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642355 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642382 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642427 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642431 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642455 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} Jul 13 18:53:52.642683 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.642476 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4caa78aa-cb18-4101-a954-663154527a0b","Type":"ContainerDied","Data":"c16a35d39294058300bcee1a4d4f73dc48de1f0dc7c8b59bd2321949c70f1456"} Jul 13 18:53:52.650734 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.650713 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.657747 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.657728 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.664126 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.664111 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.667464 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.667445 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:53:52.673067 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.671874 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:53:52.674902 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.674887 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.684220 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.684191 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.691584 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.691563 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4caa78aa-cb18-4101-a954-663154527a0b" path="/var/lib/kubelet/pods/4caa78aa-cb18-4101-a954-663154527a0b/volumes" Jul 13 18:53:52.691923 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.691905 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.698953 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.698925 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.699185 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699168 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:53:52.699254 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.699180 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.699416 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699219 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} err="failed to get container status \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" Jul 13 18:53:52.699416 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699282 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.699504 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699469 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="config-reloader" Jul 13 18:53:52.699504 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699484 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="config-reloader" Jul 13 18:53:52.699504 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.699491 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699514 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} err="failed to get container status \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699527 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699495 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="init-config-reloader" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699567 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="init-config-reloader" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699580 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699586 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy" Jul 13 18:53:52.699607 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699608 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="thanos-sidecar" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699614 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="thanos-sidecar" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699620 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-thanos" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699627 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-thanos" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699634 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-web" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699640 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-web" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699651 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="prometheus" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699659 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="prometheus" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699731 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="config-reloader" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699740 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-thanos" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699748 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699757 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="prometheus" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699766 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="thanos-sidecar" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699771 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa78aa-cb18-4101-a954-663154527a0b" containerName="kube-rbac-proxy-web" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.699769 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699793 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} err="failed to get container status \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" Jul 13 18:53:52.699851 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.699808 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.700460 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.700015 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.700460 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700040 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} err="failed to get container status \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" Jul 13 18:53:52.700460 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700051 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.700460 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.700231 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.700460 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700265 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} err="failed to get container status \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" Jul 13 18:53:52.700460 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700279 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.700700 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.700498 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.700700 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700515 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} err="failed to get container status \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" Jul 13 18:53:52.700700 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700527 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.700792 ip-10-0-128-27 kubenswrapper[2560]: E0713 18:53:52.700723 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.700792 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700742 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} err="failed to get container status \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" Jul 13 18:53:52.700792 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700753 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.701009 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.700992 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} err="failed to get container status \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" Jul 13 18:53:52.701045 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701009 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.701204 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701189 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} err="failed to get container status \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" Jul 13 18:53:52.701288 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701204 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.701448 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701431 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} err="failed to get container status \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" Jul 13 18:53:52.701448 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701446 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.701645 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701627 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} err="failed to get container status \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" Jul 13 18:53:52.701705 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701645 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.701849 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701831 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} err="failed to get container status \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" Jul 13 18:53:52.701889 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.701850 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.702148 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702129 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} err="failed to get container status \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" Jul 13 18:53:52.702148 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702148 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.702402 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702383 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} err="failed to get container status \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" Jul 13 18:53:52.702465 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702402 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.702605 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702587 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} err="failed to get container status \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" Jul 13 18:53:52.702653 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702605 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.702789 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702771 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} err="failed to get container status \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" Jul 13 18:53:52.702829 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702790 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.702985 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702969 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} err="failed to get container status \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" Jul 13 18:53:52.703039 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.702985 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.703218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703179 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} err="failed to get container status \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" Jul 13 18:53:52.703288 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703219 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.703439 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703423 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} err="failed to get container status \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" Jul 13 18:53:52.703480 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703439 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.703656 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703638 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} err="failed to get container status \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" Jul 13 18:53:52.703697 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703656 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.703846 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703831 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} err="failed to get container status \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" Jul 13 18:53:52.703886 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.703847 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.704017 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704003 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} err="failed to get container status \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" Jul 13 18:53:52.704063 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704017 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.704185 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704169 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} err="failed to get container status \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" Jul 13 18:53:52.704226 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704185 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.704362 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704348 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} err="failed to get container status \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" Jul 13 18:53:52.704402 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704362 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.704752 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704514 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} err="failed to get container status \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" Jul 13 18:53:52.704797 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704755 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.704922 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704906 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} err="failed to get container status \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" Jul 13 18:53:52.704975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704922 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.704975 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.704923 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.705049 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705038 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} err="failed to get container status \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" Jul 13 18:53:52.705085 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705048 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.705271 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705235 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} err="failed to get container status \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" Jul 13 18:53:52.705339 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705273 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.705507 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705477 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} err="failed to get container status \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" Jul 13 18:53:52.705507 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705501 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.705731 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705710 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} err="failed to get container status \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" Jul 13 18:53:52.705785 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705732 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.705927 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705904 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} err="failed to get container status \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" Jul 13 18:53:52.706024 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.705928 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.706177 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706158 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} err="failed to get container status \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" Jul 13 18:53:52.706224 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706179 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.706429 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706409 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} err="failed to get container status \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" Jul 13 18:53:52.706490 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706429 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.706667 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706649 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} err="failed to get container status \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" Jul 13 18:53:52.706712 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706667 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.706855 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706839 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} err="failed to get container status \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" Jul 13 18:53:52.706901 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.706855 2560 scope.go:117] "RemoveContainer" containerID="9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f" Jul 13 18:53:52.707067 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707050 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f"} err="failed to get container status \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": rpc error: code = NotFound desc = could not find container \"9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f\": container with ID starting with 9dc9fa55baa9d6916c32cacb7953623fd894c83479d4e71eced2da411572fa8f not found: ID does not exist" Jul 13 18:53:52.707115 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707067 2560 scope.go:117] "RemoveContainer" containerID="03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74" Jul 13 18:53:52.707304 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707286 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74"} err="failed to get container status \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": rpc error: code = NotFound desc = could not find container \"03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74\": container with ID starting with 03511864b7a7b4bcc6fec2fdfb3250fb04da0199b99773bffa0ccb3d6b89bb74 not found: ID does not exist" Jul 13 18:53:52.707379 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707306 2560 scope.go:117] "RemoveContainer" containerID="5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec" Jul 13 18:53:52.707526 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707500 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec"} err="failed to get container status \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": rpc error: code = NotFound desc = could not find container \"5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec\": container with ID starting with 5829f65b06f757000f1e60154435b07493e7bd1594806a3534b00f33b572f1ec not found: ID does not exist" Jul 13 18:53:52.707526 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707519 2560 scope.go:117] "RemoveContainer" containerID="c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e" Jul 13 18:53:52.707754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707735 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e"} err="failed to get container status \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": rpc error: code = NotFound desc = could not find container \"c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e\": container with ID starting with c9b694d59b8464ede48fe9899a386329df6758ee14609306b34b0e472f168a4e not found: ID does not exist" Jul 13 18:53:52.707754 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707754 2560 scope.go:117] "RemoveContainer" containerID="535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f" Jul 13 18:53:52.707987 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707961 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f"} err="failed to get container status \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": rpc error: code = NotFound desc = could not find container \"535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f\": container with ID starting with 535e4dffbc4162a02ae8eb2ff8dd58414a510067e4c069592b7fde028dc25d1f not found: ID does not exist" Jul 13 18:53:52.708072 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.707988 2560 scope.go:117] "RemoveContainer" containerID="32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150" Jul 13 18:53:52.708345 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708290 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150"} err="failed to get container status \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": rpc error: code = NotFound desc = could not find container \"32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150\": container with ID starting with 32f1a724d8f9fcf78fc14d455a63bbe6fb7998f4f71c4a16d65b0eee12a3f150 not found: ID does not exist" Jul 13 18:53:52.708345 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708316 2560 scope.go:117] "RemoveContainer" containerID="55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506" Jul 13 18:53:52.708536 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708449 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 18:53:52.708536 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708449 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 18:53:52.708763 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708537 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 18:53:52.708763 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708561 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506"} err="failed to get container status \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": rpc error: code = NotFound desc = could not find container \"55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506\": container with ID starting with 55269db1a2e4048bea74f71372bfb940da9165d883539aaa948c14901e48d506 not found: ID does not exist" Jul 13 18:53:52.708763 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708594 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8gatmmofm2q8q\"" Jul 13 18:53:52.708763 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708661 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 18:53:52.708991 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708796 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 18:53:52.708991 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708821 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 18:53:52.708991 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708828 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 18:53:52.708991 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708842 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 18:53:52.708991 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708913 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fh87d\"" Jul 13 18:53:52.708991 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.708963 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 18:53:52.709395 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.709115 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 18:53:52.709395 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.709162 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 18:53:52.709395 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.709289 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 18:53:52.712002 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.711983 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 18:53:52.715817 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.715801 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 18:53:52.723000 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.722982 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:53:52.767609 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767585 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767645 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767673 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767710 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767699 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eda3c181-4d2e-49d8-965a-f14acea86a59-config-out\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767842 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eda3c181-4d2e-49d8-965a-f14acea86a59-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767842 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767918 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767918 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767867 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767985 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.767985 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.767967 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768047 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768027 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768081 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768050 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768113 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768080 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768113 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768095 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768140 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2wt\" (UniqueName: \"kubernetes.io/projected/eda3c181-4d2e-49d8-965a-f14acea86a59-kube-api-access-4g2wt\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-web-config\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768206 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768319 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-config\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768319 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.768319 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.768286 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869152 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869191 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869218 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869211 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869421 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869234 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869421 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869284 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2wt\" (UniqueName: \"kubernetes.io/projected/eda3c181-4d2e-49d8-965a-f14acea86a59-kube-api-access-4g2wt\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869421 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-web-config\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869421 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869355 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869421 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869378 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-config\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869421 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869750 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869750 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.869750 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.869547 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.870231 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870208 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.870423 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870403 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.870563 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870546 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.870679 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eda3c181-4d2e-49d8-965a-f14acea86a59-config-out\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.870782 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eda3c181-4d2e-49d8-965a-f14acea86a59-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.870904 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870885 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.871009 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.871133 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.871099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.871259 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.871224 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.871382 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.871363 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872037 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872015 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872412 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872388 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872499 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872436 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-web-config\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872499 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872392 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-config\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872499 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.870412 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872499 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872388 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872767 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872847 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.872824 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.872910 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.871028 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.873050 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.873025 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.873675 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.873651 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.874064 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.874042 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.874955 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.874929 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.875053 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.875009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eda3c181-4d2e-49d8-965a-f14acea86a59-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.875053 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.875040 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eda3c181-4d2e-49d8-965a-f14acea86a59-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.875165 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.875056 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eda3c181-4d2e-49d8-965a-f14acea86a59-config-out\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.876165 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.876145 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eda3c181-4d2e-49d8-965a-f14acea86a59-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:52.878046 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:52.878027 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2wt\" (UniqueName: \"kubernetes.io/projected/eda3c181-4d2e-49d8-965a-f14acea86a59-kube-api-access-4g2wt\") pod \"prometheus-k8s-0\" (UID: \"eda3c181-4d2e-49d8-965a-f14acea86a59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:53.015637 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:53.015602 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:53:53.175327 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:53.175236 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 18:53:53.178488 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:53:53.178453 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda3c181_4d2e_49d8_965a_f14acea86a59.slice/crio-373131b5f1d1b25b251afc362817cfb203e5d49488289e144f2ad618f69facca WatchSource:0}: Error finding container 373131b5f1d1b25b251afc362817cfb203e5d49488289e144f2ad618f69facca: Status 404 returned error can't find the container with id 373131b5f1d1b25b251afc362817cfb203e5d49488289e144f2ad618f69facca Jul 13 18:53:53.648521 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:53.648483 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda3c181-4d2e-49d8-965a-f14acea86a59" containerID="1c7dbab647f6636ad5f781d16f80a36f2a27ea27d609b9ffa5bf0172f4f422f7" exitCode=0 Jul 13 18:53:53.648966 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:53.648561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerDied","Data":"1c7dbab647f6636ad5f781d16f80a36f2a27ea27d609b9ffa5bf0172f4f422f7"} Jul 13 18:53:53.648966 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:53.648633 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"373131b5f1d1b25b251afc362817cfb203e5d49488289e144f2ad618f69facca"} Jul 13 18:53:54.656228 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.656195 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"2e91ecea5c8ba9a4b635624f44b8988be6b75c68ce421d59db22dfae03b6c612"} Jul 13 18:53:54.656597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.656306 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"373b6775ee32fd132427a7f7cea28634a3f1a85cbf657d5c7e0c91c28bb51415"} Jul 13 18:53:54.656597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.656333 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"35d879ce0578fc507b55c585bc194ea4288363bc58812175c35173f0cc5e9334"} Jul 13 18:53:54.656597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.656356 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"9751339ba914a09a2b400616944fcfc92ea4bbde3163818026448e1d3d8294f6"} Jul 13 18:53:54.656597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.656379 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"3c5d65ffc67ac6dac3ea1a0e43bc04ec2d6c7d130d4868c61c35bbc7fcdf24f1"} Jul 13 18:53:54.656597 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.656401 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eda3c181-4d2e-49d8-965a-f14acea86a59","Type":"ContainerStarted","Data":"2c5a450f407de57d6942a8020111d676bfafd14d32ee404c756d56bc67d4577d"} Jul 13 18:53:54.687587 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:54.687523 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.668875356 podStartE2EDuration="2.687503439s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.628083ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:53:54.686161129 +0000 UTC m=+224.494867190" watchObservedRunningTime="2026-07-13 18:53:54.687503439 +0000 UTC m=+224.496209434" Jul 13 18:53:58.015841 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:53:58.015808 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:54:53.016729 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:54:53.016691 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:54:53.032106 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:54:53.032078 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:54:53.910948 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:54:53.910919 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 18:55:10.622157 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.622126 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 18:55:10.623081 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.623059 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 18:55:10.624080 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.624057 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 18:55:10.624973 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.624953 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 18:55:10.626977 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.626956 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:55:10.629069 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.629051 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 18:55:10.635738 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:55:10.635720 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 18:58:53.225377 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.225344 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-7dpqq"] Jul 13 18:58:53.228858 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.228837 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.232441 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.232423 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 13 18:58:53.232546 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.232424 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 13 18:58:53.232546 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.232422 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 13 18:58:53.233610 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.233597 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-knvzt\"" Jul 13 18:58:53.236817 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.236800 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7dpqq"] Jul 13 18:58:53.285391 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.285366 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4hc\" (UniqueName: \"kubernetes.io/projected/bca6140e-f564-4734-9264-3fd022e21ac6-kube-api-access-pf4hc\") pod \"s3-init-7dpqq\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.285517 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.285409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca6140e-f564-4734-9264-3fd022e21ac6-model-cache\") pod \"s3-init-7dpqq\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.386361 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.386337 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4hc\" (UniqueName: \"kubernetes.io/projected/bca6140e-f564-4734-9264-3fd022e21ac6-kube-api-access-pf4hc\") pod \"s3-init-7dpqq\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.386481 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.386375 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca6140e-f564-4734-9264-3fd022e21ac6-model-cache\") pod \"s3-init-7dpqq\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.386690 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.386675 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca6140e-f564-4734-9264-3fd022e21ac6-model-cache\") pod \"s3-init-7dpqq\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.395838 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.395819 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4hc\" (UniqueName: \"kubernetes.io/projected/bca6140e-f564-4734-9264-3fd022e21ac6-kube-api-access-pf4hc\") pod \"s3-init-7dpqq\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.548219 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.548194 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7dpqq" Jul 13 18:58:53.675308 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.675279 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7dpqq"] Jul 13 18:58:53.682796 ip-10-0-128-27 kubenswrapper[2560]: W0713 18:58:53.682772 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca6140e_f564_4734_9264_3fd022e21ac6.slice/crio-12eabf6f880408988914a5acee143b7d9dd85aa3a5f2935882729431bb7ed6f2 WatchSource:0}: Error finding container 12eabf6f880408988914a5acee143b7d9dd85aa3a5f2935882729431bb7ed6f2: Status 404 returned error can't find the container with id 12eabf6f880408988914a5acee143b7d9dd85aa3a5f2935882729431bb7ed6f2 Jul 13 18:58:53.684625 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.684606 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 18:58:53.844067 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:53.843987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7dpqq" event={"ID":"bca6140e-f564-4734-9264-3fd022e21ac6","Type":"ContainerStarted","Data":"12eabf6f880408988914a5acee143b7d9dd85aa3a5f2935882729431bb7ed6f2"} Jul 13 18:58:57.866966 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:57.866915 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7dpqq" event={"ID":"bca6140e-f564-4734-9264-3fd022e21ac6","Type":"ContainerStarted","Data":"5ed3bb2d470ed24b1e2b7ff005c43cbbc1ea62f02ed69f39daebc6a47d8eb381"} Jul 13 18:58:59.876579 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:59.876547 2560 generic.go:356] "Generic (PLEG): container finished" podID="bca6140e-f564-4734-9264-3fd022e21ac6" containerID="5ed3bb2d470ed24b1e2b7ff005c43cbbc1ea62f02ed69f39daebc6a47d8eb381" exitCode=0 Jul 13 18:58:59.876929 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:58:59.876618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7dpqq" event={"ID":"bca6140e-f564-4734-9264-3fd022e21ac6","Type":"ContainerDied","Data":"5ed3bb2d470ed24b1e2b7ff005c43cbbc1ea62f02ed69f39daebc6a47d8eb381"} Jul 13 18:59:04.902649 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:59:04.902613 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7dpqq" event={"ID":"bca6140e-f564-4734-9264-3fd022e21ac6","Type":"ContainerStarted","Data":"6c5541b2c1cbe2755950718343132d2d350ae6f02d102fcf89cf555bf2d5b38d"} Jul 13 18:59:04.921343 ip-10-0-128-27 kubenswrapper[2560]: I0713 18:59:04.921299 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-7dpqq" podStartSLOduration=1.9884936519999998 podStartE2EDuration="11.921285953s" totalImagesPullingTime="7.783896691s" totalInitContainerRuntime="2.14889561s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:04.920010105 +0000 UTC m=+534.728716124" watchObservedRunningTime="2026-07-13 18:59:04.921285953 +0000 UTC m=+534.729991953" Jul 13 19:00:10.169572 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.169538 2560 generic.go:356] "Generic (PLEG): container finished" podID="bca6140e-f564-4734-9264-3fd022e21ac6" containerID="6c5541b2c1cbe2755950718343132d2d350ae6f02d102fcf89cf555bf2d5b38d" exitCode=0 Jul 13 19:00:10.169985 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.169570 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7dpqq" event={"ID":"bca6140e-f564-4734-9264-3fd022e21ac6","Type":"ContainerDied","Data":"6c5541b2c1cbe2755950718343132d2d350ae6f02d102fcf89cf555bf2d5b38d"} Jul 13 19:00:10.652971 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.652943 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:00:10.653164 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.653088 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:00:10.653809 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.653795 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:00:10.653952 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.653932 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:00:10.656911 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.656893 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:00:10.657055 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:10.657041 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:00:11.292684 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.292657 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7dpqq" Jul 13 19:00:11.437894 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.437823 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca6140e-f564-4734-9264-3fd022e21ac6-model-cache\") pod \"bca6140e-f564-4734-9264-3fd022e21ac6\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " Jul 13 19:00:11.437894 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.437869 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf4hc\" (UniqueName: \"kubernetes.io/projected/bca6140e-f564-4734-9264-3fd022e21ac6-kube-api-access-pf4hc\") pod \"bca6140e-f564-4734-9264-3fd022e21ac6\" (UID: \"bca6140e-f564-4734-9264-3fd022e21ac6\") " Jul 13 19:00:11.440073 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.440049 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca6140e-f564-4734-9264-3fd022e21ac6-kube-api-access-pf4hc" (OuterVolumeSpecName: "kube-api-access-pf4hc") pod "bca6140e-f564-4734-9264-3fd022e21ac6" (UID: "bca6140e-f564-4734-9264-3fd022e21ac6"). InnerVolumeSpecName "kube-api-access-pf4hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:00:11.456366 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.456336 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca6140e-f564-4734-9264-3fd022e21ac6-model-cache" (OuterVolumeSpecName: "model-cache") pod "bca6140e-f564-4734-9264-3fd022e21ac6" (UID: "bca6140e-f564-4734-9264-3fd022e21ac6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:00:11.538996 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.538973 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca6140e-f564-4734-9264-3fd022e21ac6-model-cache\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:00:11.538996 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:11.538996 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4hc\" (UniqueName: \"kubernetes.io/projected/bca6140e-f564-4734-9264-3fd022e21ac6-kube-api-access-pf4hc\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:00:12.178044 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:12.178011 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7dpqq" Jul 13 19:00:12.178212 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:12.178009 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7dpqq" event={"ID":"bca6140e-f564-4734-9264-3fd022e21ac6","Type":"ContainerDied","Data":"12eabf6f880408988914a5acee143b7d9dd85aa3a5f2935882729431bb7ed6f2"} Jul 13 19:00:12.178212 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:12.178129 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12eabf6f880408988914a5acee143b7d9dd85aa3a5f2935882729431bb7ed6f2" Jul 13 19:00:24.954479 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.954437 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k"] Jul 13 19:00:24.954860 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.954772 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca6140e-f564-4734-9264-3fd022e21ac6" containerName="download-hf-model" Jul 13 19:00:24.954860 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.954782 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca6140e-f564-4734-9264-3fd022e21ac6" containerName="download-hf-model" Jul 13 19:00:24.954860 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.954801 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca6140e-f564-4734-9264-3fd022e21ac6" containerName="s3-init" Jul 13 19:00:24.954860 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.954807 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca6140e-f564-4734-9264-3fd022e21ac6" containerName="s3-init" Jul 13 19:00:24.954993 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.954873 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca6140e-f564-4734-9264-3fd022e21ac6" containerName="s3-init" Jul 13 19:00:24.961178 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.961146 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:24.964135 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.964112 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 13 19:00:24.964292 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.964139 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 13 19:00:24.964292 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.964112 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Jul 13 19:00:24.964292 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.964112 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Jul 13 19:00:24.964461 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.964419 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 13 19:00:24.965131 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.965114 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 13 19:00:24.965307 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.965291 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-98mz6\"" Jul 13 19:00:24.974352 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:24.974331 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k"] Jul 13 19:00:25.053220 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.053189 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.053220 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.053226 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df890e3d-097f-49f5-8c34-37048616e6d4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.053469 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.053260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df890e3d-097f-49f5-8c34-37048616e6d4-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.053469 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.053367 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.053469 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.053417 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4h5v\" (UniqueName: \"kubernetes.io/projected/df890e3d-097f-49f5-8c34-37048616e6d4-kube-api-access-x4h5v\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.153972 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.153940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.153972 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.153973 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df890e3d-097f-49f5-8c34-37048616e6d4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.154215 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.153996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df890e3d-097f-49f5-8c34-37048616e6d4-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.154215 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.154037 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.154215 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.154078 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4h5v\" (UniqueName: \"kubernetes.io/projected/df890e3d-097f-49f5-8c34-37048616e6d4-kube-api-access-x4h5v\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.154458 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.154439 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df890e3d-097f-49f5-8c34-37048616e6d4-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.154752 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.154731 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.154795 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.154739 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.156425 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.156396 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df890e3d-097f-49f5-8c34-37048616e6d4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.163703 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.163678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4h5v\" (UniqueName: \"kubernetes.io/projected/df890e3d-097f-49f5-8c34-37048616e6d4-kube-api-access-x4h5v\") pod \"isvc-xgboost-graph-predictor-555fc69d56-8z52k\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.171657 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.171636 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh"] Jul 13 19:00:25.175378 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.175360 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.178061 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.178039 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 13 19:00:25.178061 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.178043 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 13 19:00:25.185920 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.185902 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh"] Jul 13 19:00:25.255017 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.254996 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.255201 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.255027 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.255201 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.255060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca7e71e-2227-458a-9812-840ffccc956c-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.255201 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.255077 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.255201 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.255150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nft7t\" (UniqueName: \"kubernetes.io/projected/4ca7e71e-2227-458a-9812-840ffccc956c-kube-api-access-nft7t\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.271386 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.271365 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:25.355832 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.355806 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nft7t\" (UniqueName: \"kubernetes.io/projected/4ca7e71e-2227-458a-9812-840ffccc956c-kube-api-access-nft7t\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.355976 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.355909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.355976 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.355946 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.356089 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.356002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca7e71e-2227-458a-9812-840ffccc956c-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.356089 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.356028 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.356186 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:00:25.356108 2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Jul 13 19:00:25.356186 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:00:25.356175 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls podName:4ca7e71e-2227-458a-9812-840ffccc956c nodeName:}" failed. No retries permitted until 2026-07-13 19:00:25.856152305 +0000 UTC m=+615.664858286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls") pod "isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" (UID: "4ca7e71e-2227-458a-9812-840ffccc956c") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Jul 13 19:00:25.356571 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.356541 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca7e71e-2227-458a-9812-840ffccc956c-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.356734 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.356716 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.356789 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.356778 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.368552 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.368528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nft7t\" (UniqueName: \"kubernetes.io/projected/4ca7e71e-2227-458a-9812-840ffccc956c-kube-api-access-nft7t\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.405289 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.405265 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k"] Jul 13 19:00:25.407641 ip-10-0-128-27 kubenswrapper[2560]: W0713 19:00:25.407617 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf890e3d_097f_49f5_8c34_37048616e6d4.slice/crio-032fc179d0596104122085bb90d60dd0b2f2d09df0ae363a4035e1ad14753808 WatchSource:0}: Error finding container 032fc179d0596104122085bb90d60dd0b2f2d09df0ae363a4035e1ad14753808: Status 404 returned error can't find the container with id 032fc179d0596104122085bb90d60dd0b2f2d09df0ae363a4035e1ad14753808 Jul 13 19:00:25.859742 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.859715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:25.861987 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:25.861970 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:26.086398 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:26.086363 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:26.215491 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:26.215464 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh"] Jul 13 19:00:26.217295 ip-10-0-128-27 kubenswrapper[2560]: W0713 19:00:26.217274 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca7e71e_2227_458a_9812_840ffccc956c.slice/crio-a55848268324ac2cdedd7134aad3133694ba0e33bbb56799dc35d090b11b8265 WatchSource:0}: Error finding container a55848268324ac2cdedd7134aad3133694ba0e33bbb56799dc35d090b11b8265: Status 404 returned error can't find the container with id a55848268324ac2cdedd7134aad3133694ba0e33bbb56799dc35d090b11b8265 Jul 13 19:00:26.239954 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:26.239928 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerStarted","Data":"a55848268324ac2cdedd7134aad3133694ba0e33bbb56799dc35d090b11b8265"} Jul 13 19:00:26.241568 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:26.241546 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerStarted","Data":"aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b"} Jul 13 19:00:26.241673 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:26.241603 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerStarted","Data":"032fc179d0596104122085bb90d60dd0b2f2d09df0ae363a4035e1ad14753808"} Jul 13 19:00:27.246859 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:27.246823 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerStarted","Data":"ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97"} Jul 13 19:00:29.257513 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:29.257472 2560 generic.go:356] "Generic (PLEG): container finished" podID="df890e3d-097f-49f5-8c34-37048616e6d4" containerID="aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b" exitCode=0 Jul 13 19:00:29.257921 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:29.257545 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerDied","Data":"aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b"} Jul 13 19:00:30.266676 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:30.266636 2560 generic.go:356] "Generic (PLEG): container finished" podID="4ca7e71e-2227-458a-9812-840ffccc956c" containerID="ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97" exitCode=0 Jul 13 19:00:30.267155 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:30.266745 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerDied","Data":"ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97"} Jul 13 19:00:55.407274 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:55.407167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerStarted","Data":"28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf"} Jul 13 19:00:55.409639 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:55.409580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerStarted","Data":"3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190"} Jul 13 19:00:58.426372 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.426338 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerStarted","Data":"bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2"} Jul 13 19:00:58.426829 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.426571 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:58.426829 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.426629 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:00:58.428155 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.428115 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:00:58.428711 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.428691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerStarted","Data":"4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd"} Jul 13 19:00:58.428812 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.428745 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:58.428812 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.428788 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:00:58.429786 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.429734 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:00:58.445025 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.444986 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podStartSLOduration=2.319143417 podStartE2EDuration="34.444971853s" totalImagesPullingTime="28.541224478s" totalInitContainerRuntime="3.584603958s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:58.444192132 +0000 UTC m=+648.252898158" watchObservedRunningTime="2026-07-13 19:00:58.444971853 +0000 UTC m=+648.253677853" Jul 13 19:00:58.463112 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:58.463072 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podStartSLOduration=2.39593132 podStartE2EDuration="33.46305765s" totalImagesPullingTime="27.461849753s" totalInitContainerRuntime="3.605276577s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:58.461904201 +0000 UTC m=+648.270610203" watchObservedRunningTime="2026-07-13 19:00:58.46305765 +0000 UTC m=+648.271763651" Jul 13 19:00:59.433287 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:59.433228 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:00:59.433672 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:00:59.433337 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:01:04.438299 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:04.438266 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:01:04.438722 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:04.438450 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:01:04.438722 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:04.438649 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:01:04.439094 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:04.439072 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:01:14.438867 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:14.438827 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:01:14.439262 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:14.439015 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:01:24.439101 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:24.439066 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:01:24.439101 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:24.439083 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:01:34.438660 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:34.438619 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:01:34.439089 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:34.439034 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:01:44.439053 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:44.439015 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Jul 13 19:01:44.439446 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:44.439016 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:01:54.439569 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:54.439528 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:01:54.440042 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:01:54.440026 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:02:04.439439 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:04.439358 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 13 19:02:14.439935 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:14.439899 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:02:44.680185 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.680152 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k"] Jul 13 19:02:44.680839 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.680426 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" containerID="cri-o://3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190" gracePeriod=30 Jul 13 19:02:44.680839 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.680440 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kube-rbac-proxy" containerID="cri-o://bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2" gracePeriod=30 Jul 13 19:02:44.791663 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.791630 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh"] Jul 13 19:02:44.791972 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.791943 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" containerID="cri-o://28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf" gracePeriod=30 Jul 13 19:02:44.792046 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.791985 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kube-rbac-proxy" containerID="cri-o://4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd" gracePeriod=30 Jul 13 19:02:44.872232 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.872200 2560 generic.go:356] "Generic (PLEG): container finished" podID="df890e3d-097f-49f5-8c34-37048616e6d4" containerID="bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2" exitCode=2 Jul 13 19:02:44.872391 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:44.872268 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerDied","Data":"bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2"} Jul 13 19:02:45.878726 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:45.878690 2560 generic.go:356] "Generic (PLEG): container finished" podID="4ca7e71e-2227-458a-9812-840ffccc956c" containerID="4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd" exitCode=2 Jul 13 19:02:45.879101 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:45.878760 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerDied","Data":"4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd"} Jul 13 19:02:48.129190 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.129166 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:02:48.231058 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.230958 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-cabundle-cert\") pod \"df890e3d-097f-49f5-8c34-37048616e6d4\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " Jul 13 19:02:48.231058 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231019 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-kube-rbac-proxy-sar-config\") pod \"df890e3d-097f-49f5-8c34-37048616e6d4\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " Jul 13 19:02:48.231333 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231229 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4h5v\" (UniqueName: \"kubernetes.io/projected/df890e3d-097f-49f5-8c34-37048616e6d4-kube-api-access-x4h5v\") pod \"df890e3d-097f-49f5-8c34-37048616e6d4\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " Jul 13 19:02:48.231333 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231301 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df890e3d-097f-49f5-8c34-37048616e6d4-proxy-tls\") pod \"df890e3d-097f-49f5-8c34-37048616e6d4\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " Jul 13 19:02:48.231333 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231325 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df890e3d-097f-49f5-8c34-37048616e6d4-kserve-provision-location\") pod \"df890e3d-097f-49f5-8c34-37048616e6d4\" (UID: \"df890e3d-097f-49f5-8c34-37048616e6d4\") " Jul 13 19:02:48.231333 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231322 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df890e3d-097f-49f5-8c34-37048616e6d4" (UID: "df890e3d-097f-49f5-8c34-37048616e6d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:02:48.231531 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231389 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df890e3d-097f-49f5-8c34-37048616e6d4" (UID: "df890e3d-097f-49f5-8c34-37048616e6d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:02:48.231656 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231635 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df890e3d-097f-49f5-8c34-37048616e6d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df890e3d-097f-49f5-8c34-37048616e6d4" (UID: "df890e3d-097f-49f5-8c34-37048616e6d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:02:48.231726 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231663 2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-cabundle-cert\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:48.231726 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.231686 2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df890e3d-097f-49f5-8c34-37048616e6d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:48.233399 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.233377 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df890e3d-097f-49f5-8c34-37048616e6d4-kube-api-access-x4h5v" (OuterVolumeSpecName: "kube-api-access-x4h5v") pod "df890e3d-097f-49f5-8c34-37048616e6d4" (UID: "df890e3d-097f-49f5-8c34-37048616e6d4"). InnerVolumeSpecName "kube-api-access-x4h5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:02:48.233399 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.233401 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df890e3d-097f-49f5-8c34-37048616e6d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df890e3d-097f-49f5-8c34-37048616e6d4" (UID: "df890e3d-097f-49f5-8c34-37048616e6d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:02:48.332524 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.332481 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4h5v\" (UniqueName: \"kubernetes.io/projected/df890e3d-097f-49f5-8c34-37048616e6d4-kube-api-access-x4h5v\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:48.332524 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.332521 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df890e3d-097f-49f5-8c34-37048616e6d4-proxy-tls\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:48.332524 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.332533 2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df890e3d-097f-49f5-8c34-37048616e6d4-kserve-provision-location\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:48.893897 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.893861 2560 generic.go:356] "Generic (PLEG): container finished" podID="df890e3d-097f-49f5-8c34-37048616e6d4" containerID="3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190" exitCode=0 Jul 13 19:02:48.893897 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.893900 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerDied","Data":"3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190"} Jul 13 19:02:48.894143 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.893947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" event={"ID":"df890e3d-097f-49f5-8c34-37048616e6d4","Type":"ContainerDied","Data":"032fc179d0596104122085bb90d60dd0b2f2d09df0ae363a4035e1ad14753808"} Jul 13 19:02:48.894143 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.893963 2560 scope.go:117] "RemoveContainer" containerID="bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2" Jul 13 19:02:48.894143 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.893946 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k" Jul 13 19:02:48.902321 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.902229 2560 scope.go:117] "RemoveContainer" containerID="3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190" Jul 13 19:02:48.909756 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.909738 2560 scope.go:117] "RemoveContainer" containerID="aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b" Jul 13 19:02:48.912865 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.912844 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k"] Jul 13 19:02:48.916325 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.916302 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-8z52k"] Jul 13 19:02:48.917827 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.917792 2560 scope.go:117] "RemoveContainer" containerID="bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2" Jul 13 19:02:48.918041 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:02:48.918022 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2\": container with ID starting with bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2 not found: ID does not exist" containerID="bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2" Jul 13 19:02:48.918089 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.918047 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2"} err="failed to get container status \"bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2\": rpc error: code = NotFound desc = could not find container \"bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2\": container with ID starting with bb09758222bfa2bc68a1ab8ca350776999f12d9d918d8edeb46070a018306ad2 not found: ID does not exist" Jul 13 19:02:48.918089 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.918067 2560 scope.go:117] "RemoveContainer" containerID="3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190" Jul 13 19:02:48.918323 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:02:48.918303 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190\": container with ID starting with 3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190 not found: ID does not exist" containerID="3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190" Jul 13 19:02:48.918370 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.918332 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190"} err="failed to get container status \"3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190\": rpc error: code = NotFound desc = could not find container \"3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190\": container with ID starting with 3072ee2a3408c6b59509121a9dae59bd123d427011ef51213931470aad19f190 not found: ID does not exist" Jul 13 19:02:48.918370 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.918353 2560 scope.go:117] "RemoveContainer" containerID="aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b" Jul 13 19:02:48.918580 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:02:48.918561 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b\": container with ID starting with aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b not found: ID does not exist" containerID="aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b" Jul 13 19:02:48.918631 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:48.918588 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b"} err="failed to get container status \"aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b\": rpc error: code = NotFound desc = could not find container \"aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b\": container with ID starting with aea8a875bbeabc909a80c6caadb993160ab8705dc35489244d97c2dbee47d21b not found: ID does not exist" Jul 13 19:02:49.430065 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.430038 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:02:49.545184 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545100 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nft7t\" (UniqueName: \"kubernetes.io/projected/4ca7e71e-2227-458a-9812-840ffccc956c-kube-api-access-nft7t\") pod \"4ca7e71e-2227-458a-9812-840ffccc956c\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " Jul 13 19:02:49.545184 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545160 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls\") pod \"4ca7e71e-2227-458a-9812-840ffccc956c\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " Jul 13 19:02:49.545450 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545190 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-cabundle-cert\") pod \"4ca7e71e-2227-458a-9812-840ffccc956c\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " Jul 13 19:02:49.545450 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545224 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-kube-rbac-proxy-sar-config\") pod \"4ca7e71e-2227-458a-9812-840ffccc956c\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " Jul 13 19:02:49.545450 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545271 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca7e71e-2227-458a-9812-840ffccc956c-kserve-provision-location\") pod \"4ca7e71e-2227-458a-9812-840ffccc956c\" (UID: \"4ca7e71e-2227-458a-9812-840ffccc956c\") " Jul 13 19:02:49.545631 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545605 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ca7e71e-2227-458a-9812-840ffccc956c" (UID: "4ca7e71e-2227-458a-9812-840ffccc956c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:02:49.545631 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545614 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ca7e71e-2227-458a-9812-840ffccc956c" (UID: "4ca7e71e-2227-458a-9812-840ffccc956c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:02:49.545711 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.545633 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca7e71e-2227-458a-9812-840ffccc956c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ca7e71e-2227-458a-9812-840ffccc956c" (UID: "4ca7e71e-2227-458a-9812-840ffccc956c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:02:49.547226 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.547206 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca7e71e-2227-458a-9812-840ffccc956c-kube-api-access-nft7t" (OuterVolumeSpecName: "kube-api-access-nft7t") pod "4ca7e71e-2227-458a-9812-840ffccc956c" (UID: "4ca7e71e-2227-458a-9812-840ffccc956c"). InnerVolumeSpecName "kube-api-access-nft7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:02:49.547298 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.547254 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ca7e71e-2227-458a-9812-840ffccc956c" (UID: "4ca7e71e-2227-458a-9812-840ffccc956c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:02:49.646391 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.646355 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nft7t\" (UniqueName: \"kubernetes.io/projected/4ca7e71e-2227-458a-9812-840ffccc956c-kube-api-access-nft7t\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:49.646391 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.646382 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca7e71e-2227-458a-9812-840ffccc956c-proxy-tls\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:49.646391 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.646397 2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-cabundle-cert\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:49.646613 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.646409 2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca7e71e-2227-458a-9812-840ffccc956c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:49.646613 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.646420 2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca7e71e-2227-458a-9812-840ffccc956c-kserve-provision-location\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:02:49.900156 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.900061 2560 generic.go:356] "Generic (PLEG): container finished" podID="4ca7e71e-2227-458a-9812-840ffccc956c" containerID="28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf" exitCode=0 Jul 13 19:02:49.900156 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.900120 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" Jul 13 19:02:49.900156 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.900144 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerDied","Data":"28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf"} Jul 13 19:02:49.900446 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.900199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh" event={"ID":"4ca7e71e-2227-458a-9812-840ffccc956c","Type":"ContainerDied","Data":"a55848268324ac2cdedd7134aad3133694ba0e33bbb56799dc35d090b11b8265"} Jul 13 19:02:49.900446 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.900227 2560 scope.go:117] "RemoveContainer" containerID="4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd" Jul 13 19:02:49.908623 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.908586 2560 scope.go:117] "RemoveContainer" containerID="28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf" Jul 13 19:02:49.915837 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.915821 2560 scope.go:117] "RemoveContainer" containerID="ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97" Jul 13 19:02:49.921884 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.921864 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh"] Jul 13 19:02:49.923126 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.923112 2560 scope.go:117] "RemoveContainer" containerID="4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd" Jul 13 19:02:49.923390 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:02:49.923371 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd\": container with ID starting with 4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd not found: ID does not exist" containerID="4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd" Jul 13 19:02:49.923465 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.923402 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd"} err="failed to get container status \"4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd\": rpc error: code = NotFound desc = could not find container \"4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd\": container with ID starting with 4781eaa30b3c78fc820333f69d7343719a1cff9e68092f57eea22f6cfe4af0dd not found: ID does not exist" Jul 13 19:02:49.923465 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.923424 2560 scope.go:117] "RemoveContainer" containerID="28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf" Jul 13 19:02:49.923753 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:02:49.923726 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf\": container with ID starting with 28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf not found: ID does not exist" containerID="28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf" Jul 13 19:02:49.923857 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.923758 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf"} err="failed to get container status \"28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf\": rpc error: code = NotFound desc = could not find container \"28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf\": container with ID starting with 28fde8f6953958fd313a1dc6aeb5743c37b62852cc8c738fd5dcc0c19ac030bf not found: ID does not exist" Jul 13 19:02:49.923857 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.923775 2560 scope.go:117] "RemoveContainer" containerID="ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97" Jul 13 19:02:49.924235 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:02:49.924210 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97\": container with ID starting with ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97 not found: ID does not exist" containerID="ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97" Jul 13 19:02:49.924351 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.924260 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97"} err="failed to get container status \"ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97\": rpc error: code = NotFound desc = could not find container \"ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97\": container with ID starting with ca3c2e0d4fe3655e81979915c723ac41cda698ec6de77c564ab8c2db0abe3b97 not found: ID does not exist" Jul 13 19:02:49.925997 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:49.925973 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d85b9cf8d-xxkwh"] Jul 13 19:02:50.690704 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:50.690672 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" path="/var/lib/kubelet/pods/4ca7e71e-2227-458a-9812-840ffccc956c/volumes" Jul 13 19:02:50.691163 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:02:50.691149 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" path="/var/lib/kubelet/pods/df890e3d-097f-49f5-8c34-37048616e6d4/volumes" Jul 13 19:05:10.679656 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:05:10.679582 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:05:10.680629 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:05:10.680605 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:05:10.682111 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:05:10.682092 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:05:10.683058 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:05:10.683038 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:05:10.684098 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:05:10.684080 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:05:10.686282 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:05:10.686263 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:10:10.706382 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:10:10.706353 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:10:10.708945 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:10:10.707079 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:10:10.709885 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:10:10.709861 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:10:10.710478 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:10:10.710460 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:10:10.710729 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:10:10.710712 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:10:10.714229 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:10:10.714203 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:15:10.732721 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:15:10.732616 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:15:10.736954 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:15:10.733440 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:15:10.736954 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:15:10.736816 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:15:10.737852 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:15:10.737834 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:15:10.738732 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:15:10.738715 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:15:10.742135 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:15:10.742118 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:20:10.759886 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:20:10.759854 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:20:10.762193 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:20:10.760768 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:20:10.764140 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:20:10.764120 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:20:10.765158 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:20:10.765140 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:20:10.766005 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:20:10.765987 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:20:10.769409 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:20:10.769391 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:25:10.785954 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:25:10.785842 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:25:10.790287 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:25:10.786749 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:25:10.790287 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:25:10.790120 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:25:10.792378 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:25:10.792359 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:25:10.793093 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:25:10.793074 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:25:10.796426 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:25:10.796410 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:30:10.812063 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:30:10.811947 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:30:10.816418 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:30:10.812861 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:30:10.816418 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:30:10.816042 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:30:10.819098 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:30:10.819074 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:30:10.820107 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:30:10.820090 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:30:10.823375 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:30:10.823360 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:35:10.839845 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:35:10.839752 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:35:10.844202 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:35:10.840673 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:35:10.844300 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:35:10.844218 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:35:10.847956 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:35:10.847921 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:35:10.848842 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:35:10.848826 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:35:10.851801 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:35:10.851785 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:40:10.865679 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:10.865581 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:40:10.869813 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:10.866548 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:40:10.870193 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:10.870177 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:40:10.879996 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:10.879978 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:40:10.880851 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:10.880835 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:40:10.884085 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:10.884064 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:40:14.939205 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939172 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gwt5m/must-gather-sf49b"] Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939502 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="storage-initializer" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939513 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="storage-initializer" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939522 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939529 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939535 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kube-rbac-proxy" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939540 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kube-rbac-proxy" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939551 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939555 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939563 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kube-rbac-proxy" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939567 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kube-rbac-proxy" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939574 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="storage-initializer" Jul 13 19:40:14.939598 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939579 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="storage-initializer" Jul 13 19:40:14.939986 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939650 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kube-rbac-proxy" Jul 13 19:40:14.939986 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939659 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kserve-container" Jul 13 19:40:14.939986 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939666 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca7e71e-2227-458a-9812-840ffccc956c" containerName="kserve-container" Jul 13 19:40:14.939986 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.939674 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df890e3d-097f-49f5-8c34-37048616e6d4" containerName="kube-rbac-proxy" Jul 13 19:40:14.942703 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.942685 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:14.945496 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.945478 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gwt5m\"/\"default-dockercfg-ng8wl\"" Jul 13 19:40:14.946648 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.946630 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gwt5m\"/\"kube-root-ca.crt\"" Jul 13 19:40:14.946720 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.946630 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gwt5m\"/\"openshift-service-ca.crt\"" Jul 13 19:40:14.950327 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:14.950274 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gwt5m/must-gather-sf49b"] Jul 13 19:40:15.022441 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.022400 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-must-gather-output\") pod \"must-gather-sf49b\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.022606 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.022491 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29p4d\" (UniqueName: \"kubernetes.io/projected/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-kube-api-access-29p4d\") pod \"must-gather-sf49b\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.123675 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.123642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29p4d\" (UniqueName: \"kubernetes.io/projected/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-kube-api-access-29p4d\") pod \"must-gather-sf49b\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.123791 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.123705 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-must-gather-output\") pod \"must-gather-sf49b\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.124009 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.123995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-must-gather-output\") pod \"must-gather-sf49b\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.131408 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.131382 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29p4d\" (UniqueName: \"kubernetes.io/projected/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-kube-api-access-29p4d\") pod \"must-gather-sf49b\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.265565 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.265533 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:15.391121 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.391089 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gwt5m/must-gather-sf49b"] Jul 13 19:40:15.393930 ip-10-0-128-27 kubenswrapper[2560]: W0713 19:40:15.393903 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4fecf0b_0ff3_4205_a38b_e1afab69e2df.slice/crio-87b14bf41524519ead0ebc4edde06449f55f32f09238f009cf48ca816ca4cff0 WatchSource:0}: Error finding container 87b14bf41524519ead0ebc4edde06449f55f32f09238f009cf48ca816ca4cff0: Status 404 returned error can't find the container with id 87b14bf41524519ead0ebc4edde06449f55f32f09238f009cf48ca816ca4cff0 Jul 13 19:40:15.395676 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.395659 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 19:40:15.869167 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:15.869136 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gwt5m/must-gather-sf49b" event={"ID":"a4fecf0b-0ff3-4205-a38b-e1afab69e2df","Type":"ContainerStarted","Data":"87b14bf41524519ead0ebc4edde06449f55f32f09238f009cf48ca816ca4cff0"} Jul 13 19:40:21.901621 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:21.901570 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gwt5m/must-gather-sf49b" event={"ID":"a4fecf0b-0ff3-4205-a38b-e1afab69e2df","Type":"ContainerStarted","Data":"b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50"} Jul 13 19:40:22.907586 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:22.907548 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gwt5m/must-gather-sf49b" event={"ID":"a4fecf0b-0ff3-4205-a38b-e1afab69e2df","Type":"ContainerStarted","Data":"56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70"} Jul 13 19:40:22.923467 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:22.923416 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gwt5m/must-gather-sf49b" podStartSLOduration=2.546107036 podStartE2EDuration="8.92339934s" totalImagesPullingTime="6.377292304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:40:22.922962919 +0000 UTC m=+3012.731668923" watchObservedRunningTime="2026-07-13 19:40:22.92339934 +0000 UTC m=+3012.732105340" Jul 13 19:40:45.005535 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:45.005501 2560 generic.go:356] "Generic (PLEG): container finished" podID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerID="b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50" exitCode=0 Jul 13 19:40:45.005941 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:45.005577 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gwt5m/must-gather-sf49b" event={"ID":"a4fecf0b-0ff3-4205-a38b-e1afab69e2df","Type":"ContainerDied","Data":"b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50"} Jul 13 19:40:45.005941 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:45.005915 2560 scope.go:117] "RemoveContainer" containerID="b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50" Jul 13 19:40:45.616201 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:45.616167 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gwt5m_must-gather-sf49b_a4fecf0b-0ff3-4205-a38b-e1afab69e2df/gather/0.log" Jul 13 19:40:46.261412 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.261378 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qwzz7/must-gather-9mgvj"] Jul 13 19:40:46.264749 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.264729 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.272459 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.272431 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qwzz7\"/\"default-dockercfg-6c7xb\"" Jul 13 19:40:46.273927 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.273912 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qwzz7\"/\"kube-root-ca.crt\"" Jul 13 19:40:46.273981 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.273963 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qwzz7\"/\"openshift-service-ca.crt\"" Jul 13 19:40:46.295781 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.295762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwzz7/must-gather-9mgvj"] Jul 13 19:40:46.299985 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.299953 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7g5\" (UniqueName: \"kubernetes.io/projected/75656ad1-e01c-495a-9f45-debc7c6491e9-kube-api-access-pd7g5\") pod \"must-gather-9mgvj\" (UID: \"75656ad1-e01c-495a-9f45-debc7c6491e9\") " pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.300090 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.300000 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75656ad1-e01c-495a-9f45-debc7c6491e9-must-gather-output\") pod \"must-gather-9mgvj\" (UID: \"75656ad1-e01c-495a-9f45-debc7c6491e9\") " pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.400641 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.400610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75656ad1-e01c-495a-9f45-debc7c6491e9-must-gather-output\") pod \"must-gather-9mgvj\" (UID: \"75656ad1-e01c-495a-9f45-debc7c6491e9\") " pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.400845 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.400738 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7g5\" (UniqueName: \"kubernetes.io/projected/75656ad1-e01c-495a-9f45-debc7c6491e9-kube-api-access-pd7g5\") pod \"must-gather-9mgvj\" (UID: \"75656ad1-e01c-495a-9f45-debc7c6491e9\") " pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.400951 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.400930 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75656ad1-e01c-495a-9f45-debc7c6491e9-must-gather-output\") pod \"must-gather-9mgvj\" (UID: \"75656ad1-e01c-495a-9f45-debc7c6491e9\") " pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.413315 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.413282 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7g5\" (UniqueName: \"kubernetes.io/projected/75656ad1-e01c-495a-9f45-debc7c6491e9-kube-api-access-pd7g5\") pod \"must-gather-9mgvj\" (UID: \"75656ad1-e01c-495a-9f45-debc7c6491e9\") " pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.573582 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.573486 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwzz7/must-gather-9mgvj" Jul 13 19:40:46.704723 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:46.704580 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwzz7/must-gather-9mgvj"] Jul 13 19:40:46.707310 ip-10-0-128-27 kubenswrapper[2560]: W0713 19:40:46.707290 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75656ad1_e01c_495a_9f45_debc7c6491e9.slice/crio-948f825d7d592823f9080769d78d5a8a06221a6bc190e1189c93cbc150770736 WatchSource:0}: Error finding container 948f825d7d592823f9080769d78d5a8a06221a6bc190e1189c93cbc150770736: Status 404 returned error can't find the container with id 948f825d7d592823f9080769d78d5a8a06221a6bc190e1189c93cbc150770736 Jul 13 19:40:47.014500 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:47.014468 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/must-gather-9mgvj" event={"ID":"75656ad1-e01c-495a-9f45-debc7c6491e9","Type":"ContainerStarted","Data":"948f825d7d592823f9080769d78d5a8a06221a6bc190e1189c93cbc150770736"} Jul 13 19:40:48.020580 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:48.020544 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/must-gather-9mgvj" event={"ID":"75656ad1-e01c-495a-9f45-debc7c6491e9","Type":"ContainerStarted","Data":"b4b5f62b9563df092d41deb17569ed7792b0cc9028124c48b08b0414c0ca02af"} Jul 13 19:40:48.021049 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:48.020591 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/must-gather-9mgvj" event={"ID":"75656ad1-e01c-495a-9f45-debc7c6491e9","Type":"ContainerStarted","Data":"38ef87ca5f3eb9356e79fc1bb1131c3d3da8174f2a43bc87907ba4f3edf16939"} Jul 13 19:40:48.044335 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:48.044271 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qwzz7/must-gather-9mgvj" podStartSLOduration=1.163143248 podStartE2EDuration="2.044229206s" totalImagesPullingTime="881.085958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:40:48.043110244 +0000 UTC m=+3037.851816290" watchObservedRunningTime="2026-07-13 19:40:48.044229206 +0000 UTC m=+3037.852935208" Jul 13 19:40:49.066590 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:49.066560 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x6lrk_f21d64c2-eeba-4c25-92df-a13d3968ac4a/global-pull-secret-syncer/0.log" Jul 13 19:40:49.209388 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:49.209357 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zzhs5_62eaf7ee-ca92-4910-800b-e59a57f2776e/konnectivity-agent/0.log" Jul 13 19:40:49.237368 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:49.237341 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-27.ec2.internal_eed0f0b6c98d844417e6646ccde34c95/haproxy/0.log" Jul 13 19:40:51.048997 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.048953 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-gwt5m/must-gather-sf49b"] Jul 13 19:40:51.049548 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.049229 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-gwt5m/must-gather-sf49b" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="copy" containerID="cri-o://56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70" gracePeriod=2 Jul 13 19:40:51.051642 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:40:51.051603 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sf49b\" is forbidden: User \"system:node:ip-10-0-128-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gwt5m\": no relationship found between node 'ip-10-0-128-27.ec2.internal' and this object" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:51.053447 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.053422 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-gwt5m/must-gather-sf49b"] Jul 13 19:40:51.401258 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.397219 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gwt5m_must-gather-sf49b_a4fecf0b-0ff3-4205-a38b-e1afab69e2df/copy/0.log" Jul 13 19:40:51.401258 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.400215 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:51.405262 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:40:51.402475 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sf49b\" is forbidden: User \"system:node:ip-10-0-128-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gwt5m\": no relationship found between node 'ip-10-0-128-27.ec2.internal' and this object" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:51.457264 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.455180 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-must-gather-output\") pod \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " Jul 13 19:40:51.457264 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.455271 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29p4d\" (UniqueName: \"kubernetes.io/projected/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-kube-api-access-29p4d\") pod \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\" (UID: \"a4fecf0b-0ff3-4205-a38b-e1afab69e2df\") " Jul 13 19:40:51.460845 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.459341 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a4fecf0b-0ff3-4205-a38b-e1afab69e2df" (UID: "a4fecf0b-0ff3-4205-a38b-e1afab69e2df"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:40:51.471257 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.469469 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-kube-api-access-29p4d" (OuterVolumeSpecName: "kube-api-access-29p4d") pod "a4fecf0b-0ff3-4205-a38b-e1afab69e2df" (UID: "a4fecf0b-0ff3-4205-a38b-e1afab69e2df"). InnerVolumeSpecName "kube-api-access-29p4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:40:51.556374 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.556336 2560 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-must-gather-output\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:40:51.556374 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:51.556373 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29p4d\" (UniqueName: \"kubernetes.io/projected/a4fecf0b-0ff3-4205-a38b-e1afab69e2df-kube-api-access-29p4d\") on node \"ip-10-0-128-27.ec2.internal\" DevicePath \"\"" Jul 13 19:40:52.044638 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.044606 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gwt5m_must-gather-sf49b_a4fecf0b-0ff3-4205-a38b-e1afab69e2df/copy/0.log" Jul 13 19:40:52.045078 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.045051 2560 generic.go:356] "Generic (PLEG): container finished" podID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerID="56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70" exitCode=143 Jul 13 19:40:52.045175 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.045146 2560 scope.go:117] "RemoveContainer" containerID="56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70" Jul 13 19:40:52.045360 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.045344 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:52.052915 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:40:52.052880 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sf49b\" is forbidden: User \"system:node:ip-10-0-128-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gwt5m\": no relationship found between node 'ip-10-0-128-27.ec2.internal' and this object" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:52.063193 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:40:52.063157 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sf49b\" is forbidden: User \"system:node:ip-10-0-128-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gwt5m\": no relationship found between node 'ip-10-0-128-27.ec2.internal' and this object" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" pod="openshift-must-gather-gwt5m/must-gather-sf49b" Jul 13 19:40:52.067398 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.066994 2560 scope.go:117] "RemoveContainer" containerID="b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50" Jul 13 19:40:52.093662 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.093635 2560 scope.go:117] "RemoveContainer" containerID="56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70" Jul 13 19:40:52.094331 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:40:52.094306 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70\": container with ID starting with 56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70 not found: ID does not exist" containerID="56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70" Jul 13 19:40:52.094491 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.094466 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70"} err="failed to get container status \"56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70\": rpc error: code = NotFound desc = could not find container \"56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70\": container with ID starting with 56fc22ec61a523fa1e0eb535463aafe1c8c966098717b5b5b2702bc7bd62df70 not found: ID does not exist" Jul 13 19:40:52.094645 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.094630 2560 scope.go:117] "RemoveContainer" containerID="b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50" Jul 13 19:40:52.095191 ip-10-0-128-27 kubenswrapper[2560]: E0713 19:40:52.095151 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50\": container with ID starting with b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50 not found: ID does not exist" containerID="b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50" Jul 13 19:40:52.095301 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.095200 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50"} err="failed to get container status \"b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50\": rpc error: code = NotFound desc = could not find container \"b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50\": container with ID starting with b5727f511e58eac1027e9a316b79d52db5065cb92db09199ac6d44ccff2c2c50 not found: ID does not exist" Jul 13 19:40:52.694466 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:52.694428 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" path="/var/lib/kubelet/pods/a4fecf0b-0ff3-4205-a38b-e1afab69e2df/volumes" Jul 13 19:40:53.871654 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:53.871613 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/1.log" Jul 13 19:40:54.042713 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.042611 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7nbqm_5eb9e185-bf02-496f-a7c1-553e7abe225a/cluster-monitoring-operator/0.log" Jul 13 19:40:54.072362 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.072330 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4ldfb_17a3af64-6172-479a-bf5c-2a50b624546a/kube-state-metrics/0.log" Jul 13 19:40:54.106305 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.106219 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4ldfb_17a3af64-6172-479a-bf5c-2a50b624546a/kube-rbac-proxy-main/0.log" Jul 13 19:40:54.132661 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.132582 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4ldfb_17a3af64-6172-479a-bf5c-2a50b624546a/kube-rbac-proxy-self/0.log" Jul 13 19:40:54.164592 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.164550 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6646549cdd-m8z9p_710dc3cc-7e3f-4bf3-94b9-79c1ca18b517/metrics-server/0.log" Jul 13 19:40:54.190911 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.190881 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-t2zjg_406cc36e-9b15-4f1f-ae67-be9cd23f3007/monitoring-plugin/0.log" Jul 13 19:40:54.374712 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.374662 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scltw_56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a/node-exporter/0.log" Jul 13 19:40:54.397714 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.397628 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scltw_56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a/kube-rbac-proxy/0.log" Jul 13 19:40:54.421486 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.421458 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scltw_56c2bb7b-dc60-476a-bbe6-f8c369f5ed2a/init-textfile/0.log" Jul 13 19:40:54.453424 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.453399 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6zkxn_97602007-12cc-41fe-8d23-e6f91a2c4d42/kube-rbac-proxy-main/0.log" Jul 13 19:40:54.481475 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.481442 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6zkxn_97602007-12cc-41fe-8d23-e6f91a2c4d42/kube-rbac-proxy-self/0.log" Jul 13 19:40:54.507024 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.506994 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6zkxn_97602007-12cc-41fe-8d23-e6f91a2c4d42/openshift-state-metrics/0.log" Jul 13 19:40:54.554883 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.554848 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/prometheus/0.log" Jul 13 19:40:54.575060 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.575030 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/config-reloader/0.log" Jul 13 19:40:54.599983 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.599950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/thanos-sidecar/0.log" Jul 13 19:40:54.622983 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.622950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/kube-rbac-proxy-web/0.log" Jul 13 19:40:54.647205 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.647157 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/kube-rbac-proxy/0.log" Jul 13 19:40:54.675190 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.675106 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/kube-rbac-proxy-thanos/0.log" Jul 13 19:40:54.700978 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.700951 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eda3c181-4d2e-49d8-965a-f14acea86a59/init-config-reloader/0.log" Jul 13 19:40:54.784297 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.784267 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-6pwls_a31c45ef-7f94-473b-ac36-f42e2e8c2509/prometheus-operator-admission-webhook/0.log" Jul 13 19:40:54.814175 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.814140 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8d48d6bb-l4z5c_ddbad9e6-fcf2-40b6-8f84-e63e81450972/telemeter-client/0.log" Jul 13 19:40:54.847055 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.847017 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8d48d6bb-l4z5c_ddbad9e6-fcf2-40b6-8f84-e63e81450972/reload/0.log" Jul 13 19:40:54.876937 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.876905 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8d48d6bb-l4z5c_ddbad9e6-fcf2-40b6-8f84-e63e81450972/kube-rbac-proxy/0.log" Jul 13 19:40:54.921599 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.921566 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/thanos-query/0.log" Jul 13 19:40:54.960775 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.960740 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy-web/0.log" Jul 13 19:40:54.997152 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:54.997126 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy/0.log" Jul 13 19:40:55.031389 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:55.031363 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/prom-label-proxy/0.log" Jul 13 19:40:55.066838 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:55.066813 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy-rules/0.log" Jul 13 19:40:55.103753 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:55.103725 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696c94d74d-fczz9_5ceb26f4-f49c-4a92-8b9e-cdabf6d093a7/kube-rbac-proxy-metrics/0.log" Jul 13 19:40:56.764543 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:56.764499 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/2.log" Jul 13 19:40:56.772272 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:56.772218 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-8n96k_9183f09e-ed22-4361-bf92-d618d4a13584/console-operator/3.log" Jul 13 19:40:57.127588 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.127505 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr"] Jul 13 19:40:57.127927 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.127910 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="copy" Jul 13 19:40:57.128008 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.127930 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="copy" Jul 13 19:40:57.128008 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.127944 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="gather" Jul 13 19:40:57.128008 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.127954 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="gather" Jul 13 19:40:57.128164 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.128068 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="copy" Jul 13 19:40:57.128164 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.128085 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4fecf0b-0ff3-4205-a38b-e1afab69e2df" containerName="gather" Jul 13 19:40:57.133639 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.133610 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.142200 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.142171 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr"] Jul 13 19:40:57.216513 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.216468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-sys\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.216689 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.216524 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-podres\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.216689 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.216562 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-lib-modules\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.216689 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.216591 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggb7\" (UniqueName: \"kubernetes.io/projected/5ad5136a-b974-483f-8cca-b665b1e59112-kube-api-access-lggb7\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.216689 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.216638 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-proc\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.240417 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.240387 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-hs6ww_13c22c75-d2bc-4ab9-9a15-10d91df7f2d7/download-server/0.log" Jul 13 19:40:57.317922 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.317870 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggb7\" (UniqueName: \"kubernetes.io/projected/5ad5136a-b974-483f-8cca-b665b1e59112-kube-api-access-lggb7\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318134 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.317958 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-proc\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318134 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-sys\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318134 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318103 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-podres\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318329 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-lib-modules\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318329 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318319 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-lib-modules\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318435 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318340 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-proc\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318435 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318345 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-sys\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.318435 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.318403 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ad5136a-b974-483f-8cca-b665b1e59112-podres\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.326212 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.326191 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggb7\" (UniqueName: \"kubernetes.io/projected/5ad5136a-b974-483f-8cca-b665b1e59112-kube-api-access-lggb7\") pod \"perf-node-gather-daemonset-hqzgr\" (UID: \"5ad5136a-b974-483f-8cca-b665b1e59112\") " pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.448176 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.448103 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:57.621724 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:57.621464 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr"] Jul 13 19:40:57.624894 ip-10-0-128-27 kubenswrapper[2560]: W0713 19:40:57.624868 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ad5136a_b974_483f_8cca_b665b1e59112.slice/crio-105e98c8bbb87c8873781a5f211f5e1d620e6ee1e7427534116430f505cbf3f6 WatchSource:0}: Error finding container 105e98c8bbb87c8873781a5f211f5e1d620e6ee1e7427534116430f505cbf3f6: Status 404 returned error can't find the container with id 105e98c8bbb87c8873781a5f211f5e1d620e6ee1e7427534116430f505cbf3f6 Jul 13 19:40:58.084426 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.084394 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" event={"ID":"5ad5136a-b974-483f-8cca-b665b1e59112","Type":"ContainerStarted","Data":"ed325132b725f2e0b7587c9ed44a68631c493981f7a11b4787dbc9f69eef3dbd"} Jul 13 19:40:58.084858 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.084455 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" event={"ID":"5ad5136a-b974-483f-8cca-b665b1e59112","Type":"ContainerStarted","Data":"105e98c8bbb87c8873781a5f211f5e1d620e6ee1e7427534116430f505cbf3f6"} Jul 13 19:40:58.084858 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.084639 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:40:58.103044 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.102985 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" podStartSLOduration=1.102965502 podStartE2EDuration="1.102965502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:40:58.101305306 +0000 UTC m=+3047.910011308" watchObservedRunningTime="2026-07-13 19:40:58.102965502 +0000 UTC m=+3047.911671505" Jul 13 19:40:58.731537 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.731503 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zt6gg_7657d29c-5476-4bd5-9899-ea335ae194e9/dns/0.log" Jul 13 19:40:58.754296 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.754269 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zt6gg_7657d29c-5476-4bd5-9899-ea335ae194e9/kube-rbac-proxy/0.log" Jul 13 19:40:58.781596 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:58.781576 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pjjf_43347716-0baa-494f-9649-aeea5a817ac6/dns-node-resolver/0.log" Jul 13 19:40:59.354445 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:40:59.354418 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-trm8k_7b6dacaf-10a8-49ca-bfeb-1ddf56986535/node-ca/0.log" Jul 13 19:41:00.515998 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:00.515963 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l6mm_105d01c1-59f9-416e-b67b-4a789d295814/serve-healthcheck-canary/0.log" Jul 13 19:41:00.926174 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:00.926076 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-8w9l7_2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2/insights-operator/0.log" Jul 13 19:41:00.930378 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:00.930345 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-8w9l7_2c3f7de3-fa2b-44ef-b951-f4512f9bbdb2/insights-operator/1.log" Jul 13 19:41:00.951878 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:00.951857 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wfw4_7fa2c82b-77db-4857-bbd2-61544468ab6c/kube-rbac-proxy/0.log" Jul 13 19:41:00.977776 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:00.977751 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wfw4_7fa2c82b-77db-4857-bbd2-61544468ab6c/exporter/0.log" Jul 13 19:41:01.005335 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:01.005315 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wfw4_7fa2c82b-77db-4857-bbd2-61544468ab6c/extractor/0.log" Jul 13 19:41:03.523071 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:03.522991 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7dpqq_bca6140e-f564-4734-9264-3fd022e21ac6/s3-init/0.log" Jul 13 19:41:03.540681 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:03.540654 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7dpqq_bca6140e-f564-4734-9264-3fd022e21ac6/download-hf-model/0.log" Jul 13 19:41:04.100457 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:04.100422 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qwzz7/perf-node-gather-daemonset-hqzgr" Jul 13 19:41:09.121756 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.121727 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/kube-multus-additional-cni-plugins/0.log" Jul 13 19:41:09.144796 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.144770 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/egress-router-binary-copy/0.log" Jul 13 19:41:09.167260 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.167214 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/cni-plugins/0.log" Jul 13 19:41:09.188046 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.188026 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/bond-cni-plugin/0.log" Jul 13 19:41:09.210891 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.210868 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/routeoverride-cni/0.log" Jul 13 19:41:09.232658 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.232636 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/whereabouts-cni-bincopy/0.log" Jul 13 19:41:09.255216 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.255191 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vlhmn_774de39e-107f-49e0-9aca-8efc914d41ee/whereabouts-cni/0.log" Jul 13 19:41:09.443726 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.443651 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ghf6p_bcc88ca5-0a67-4ea4-a976-4d1f1a86f677/kube-multus/0.log" Jul 13 19:41:09.513407 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.513374 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6mvdw_17b4d31c-1830-4dc8-8984-9032c38bcc3e/network-metrics-daemon/0.log" Jul 13 19:41:09.534852 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:09.534824 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6mvdw_17b4d31c-1830-4dc8-8984-9032c38bcc3e/kube-rbac-proxy/0.log" Jul 13 19:41:10.563222 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.563183 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-controller/0.log" Jul 13 19:41:10.580865 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.580837 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/0.log" Jul 13 19:41:10.581022 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.580881 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovn-acl-logging/1.log" Jul 13 19:41:10.601062 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.601040 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/kube-rbac-proxy-node/0.log" Jul 13 19:41:10.630427 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.630404 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/kube-rbac-proxy-ovn-metrics/0.log" Jul 13 19:41:10.647775 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.647756 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/northd/0.log" Jul 13 19:41:10.669371 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.669354 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/nbdb/0.log" Jul 13 19:41:10.693399 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.693381 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/sbdb/0.log" Jul 13 19:41:10.762768 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:10.762741 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9k97_818910c4-05c0-416f-a7cc-4feee88e85c8/ovnkube-controller/0.log" Jul 13 19:41:11.917207 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:11.917179 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2zwjj_c8e41706-6b92-4540-97b9-d2edf520c1c4/network-check-target-container/0.log" Jul 13 19:41:12.871041 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:12.871010 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9gxff_bc5ddbf7-93cd-498b-b808-de40d5c7065b/iptables-alerter/0.log" Jul 13 19:41:13.547197 ip-10-0-128-27 kubenswrapper[2560]: I0713 19:41:13.547167 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-2hmf5_bac567a5-a97c-4a54-97c7-946025cc3b08/tuned/0.log"